Paper: Tree-Planner: Efficient Close-loop Task Planning with Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
When sampling plans (action sequences) non-iteratively, an error in one action step (something that diverges from the list of executable actions) can lead to subsequent actions being erroneous, leading to a so-called "compounding effect" as highlighted in [C]. This would lead to most of the sampled plans being infeasible. How do the authors address this problem?

Author response:
We thank you for your valuable comments.

(i) In Zero-shot Planner[1], compounding errors are traditionally mitigated through heuristic evaluation functions. Our approach effectively utilizes 'grounded deciding' to achieve a similar outcome, thereby preventing compounding errors with real-time environmental observations. To further enhance this process, we have implemented three heuristic evaluation functions akin to those in Zero-shot Planner for Best First Search: a) Language-Guided BFS; b) Environment-Guided BFS; c) Hybrid BFS. The detailed introduction to each method and experimental results are presented in **Appendix C.5**. Our experimental results reinforce the efficacy of grounded deciding, demonstrating its capability to implicitly avoid compounding errors by leveraging LLMs.

(ii) As detailed in our paper's quantitative analysis in **Appendix C.3**, the number of unique plans sampled increases approximately linearly with the sampling variable N, thereby ensuring a sufficiently large search space. Furthermore, our study in **Section 5.2** illustrates that the plans generated through plan sampling maintain a high success rate. This underscores the robustness and efficiency of our plan sampling methodology.

[Reference]

[1] Huang W, Abbeel P, Pathak D, et al. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents[C]//International Conference on Machine Learning. PMLR, 2022: 9118-9147.

Reviewer comment (Evaluation):
"In terms of correction efficiency, TREE-PLANNER reduces the number of corrections by 37.99% and 40.52%, respectively:" How do you define and measure an error "correction": For instance, it could be (i) a new action insertion; (ii) an action deletion; (iii) replacing one action with another generated action. Is it evaluated irrespective of the success rate (i.e. is it only measured for plans that have succeeded in correctly executing the task or for any plan)?

Author response:
We appreciate the opportunity to clarify the definition and measurement of "error correction" in our work. In our study, a "correction" is the process where the model replaces an action that failed during execution in the environment with an alternative action. This definition aligns with the one provided by Raman et al. (2022)[1]. 

[Reference]

[1] Raman S S, Cohen V, Rosen E, et al. Planning with large language models via corrective re-prompting[J]. arXiv preprint arXiv:2211.09935, 2022.

Thanks for the opportunity to provide more details. The evaluation of the number of corrections is an effectiveness metric that is independent of the success rate. In other words, we calculate this metric for all plans, including those that successfully execute the task and those that do not. We acknowledge the need to explicitly include these details in the next version of our paper to provide clearer insights into our methodology.

Reviewer comment (Novelty):
Although the paper includes a fairly comprehensive list of related works, it does not address some of the LLM-based planning work. For instance, while 'Tree of Thoughts' is referenced, there is no mention of 'Faithful CoT' [Lyu et al., 2023] and its suggestion for LLM to produce PDDL representations to be passed to a regular Planner (which would still support online planning or replanning via restarting).

Author response:
Thank you for your valuable feedback. Integrating LLMs with traditional task planners (like PDDL planners) is an important aspect of LLM-based planning, and we will include a discussion of these papers in the next version of our manuscript. [1][2] proposes producing the PDDL description of a given problem and then leveraging a classical planner to find an optimal plan. However, such work typically relies on the existence of Domain PDDL, which is often unavailable in many scenarios. While [3] suggests using LLMs as world models to construct domain PDDLs, this process still necessitates manual intervention due to issues like LLM hallucination. Besides, this work generally assumes a full observation environment, whereas our paper focuses on the more challenging partial observation environments.

[Reference]
    
[1] Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M. and Callison-Burch, C., 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.
    
[2] Liu B, Jiang Y, Zhang X, et al. Llm+ p: Empowering large language models with optimal planning proficiency[J]. arXiv preprint arXiv:2304.11477, 2023.
    
[3] Guan L, Valmeekam K, Sreedharan S, et al. Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning[J]. arXiv preprint arXiv:2305.14909, 2023.

Reviewer comment (Experiments):
It would be good to include a discussion of 'Faithful CoT' or to justify not including it (e.g. on grounds of partial observability), provided such grounds effectively preclude adoption.

Author response:
Thank you for your valuable feedback. Integrating LLMs with traditional task planners (like PDDL planners) is an important aspect of LLM-based planning, and we will include a discussion of these papers in the next version of our manuscript. [1][2] proposes producing the PDDL description of a given problem and then leveraging a classical planner to find an optimal plan. However, such work typically relies on the existence of Domain PDDL, which is often unavailable in many scenarios. While [3] suggests using LLMs as world models to construct domain PDDLs, this process still necessitates manual intervention due to issues like LLM hallucination. Besides, this work generally assumes a full observation environment, whereas our paper focuses on the more challenging partial observation environments.

[Reference]
    
[1] Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M. and Callison-Burch, C., 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.
    
[2] Liu B, Jiang Y, Zhang X, et al. Llm+ p: Empowering large language models with optimal planning proficiency[J]. arXiv preprint arXiv:2304.11477, 2023.
    
[3] Guan L, Valmeekam K, Sreedharan S, et al. Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning[J]. arXiv preprint arXiv:2305.14909, 2023.

Reviewer comment (Theory):
What guarantees scalability to domains comprizing a larger number of actions and greater tree perplexity?

Author response:
We greatly appreciate the insightful feedback provided. In response to this question, we will provide explanations from two distinct perspectives:

(i) The stored commonsense knowledge within LLMs plays a crucial role in maintaining the tree perplexity within manageable limits. This is due to LLMs' ability to efficiently navigate and prune decision trees based on contextual understanding.

(ii) Grounded deciding can also be combined with other search algorithms to address issues of greater tree perplexity. For instance, AlphaGo[1] uses Monte Carlo Tree Search (MCTS) to handle high tree complexity. We conducted additional experiments using the Best-First Search algorithm, as detailed in Appendix C.5. 
    
[Reference]
    
[1] Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016). https://doi.org/10.1038/nature16961

Reviewer comment (Experiments):
Could you discuss how different LLM could be used (GPT-4, or LLaMA) with a similar approach and whether the prompting strategy would be affected?

Author response:
Thank you for your valuable question about the interplay between the choice of Large Language Models (LLMs) and the prompting strategy. Your question raises an important aspect of our work: the potential variance in effectiveness when employing other advanced LLMs like GPT-4 or LLaMA.

As for the influence of prompting strategy, we conducted experiments with three additional prompting strategies: (i) GPT-4 optimized prompt, (ii) code-style prompt, and (iii) chain-of-thought prompt. (more details can be found in **Additional Experiments** in **General Response**)

|                      | **EXEC ↑** | **SR ↑** | **GCR ↑** | **Cost ↓** | **No.Correction ↓** |
|----------------------|------------|----------|-----------|------------|---------------------|
| TreePlanner       | 88.26±2.47 | 41.58±3.20| 59.55±3.20 | **4.54**±0.16  | **2.04**±0.26           |
| GPT-4 Optimized Prompt | **89.52**±2.15 | 39.25±0.54 | 58.80±1.05 | 4.73±0.13 | 2.06±0.18          |
| Code-Style Prompt    | 70.70±3.24 | 37.10±0.89 | 50.74±2.73 | 6.25±0.57 | 3.12±0.31          |
| Chain-of-thought    | 89.13±2.04 | **42.02**±1.27 | **60.13**±1.89 | 5.72±0.91| 2.38±0.17          |
    
We observed that different prompting strategies yielded varying results. For instance, using a GPT-4 optimized prompt and a code-style prompt didn't significantly improve performance compared to the original prompt. In contrast, employing a chain-of-thought prompting strategy resulted in a slight performance enhancement. 

It is worth noting that code-style prompting can lead to a significant decrease in performance. This could be attributed to the relatively weaker code processing capabilities of GPT-3.5 itself. In contrast, in the experiments with ProgPrompt[1], code-style prompts demonstrated stronger performance than natural language prompts, where the experiments were conducted using Codex[2]. In contrast, in the experiments with ProgPrompt, code-style prompts demonstrated stronger performance than natural language prompts, where the experiments were conducted using Codex. As the Codex API has been discontinued, we could not conduct similar experiments on Codex to explore further. In future work, we will explore other code LLMs[3][4] for our research.

Based on these findings, we believe that the choice of prompting strategy should be compatible with the training approach of the LLM. For example, code-style prompts may be more suitable for LLMs trained on code. Additionally, for LLMs trained on structured data, such as GraphGPT[5], directly providing the symbolic graph as input may yield better results than converting it into natural language prompts as in our study. Considering GPT-4's advanced understanding and context retention abilities, it's plausible that a more complex and nuanced prompting strategy could yield better results than we've observed with GPT-3.5.

We appreciate your question for highlighting this important avenue for future research.

[Reference]

[1] Singh I, Blukis V, Mousavian A, et al. Progprompt: Generating situated robot task plans using large language models[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 11523-11530.
    
[2] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[3] Xu Y, Su H, Xing C, et al. Lemur: Harmonizing Natural Language and Code for Language Agents[J]. arXiv preprint arXiv:2310.06830, 2023.

[4] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.
    
[5] Tang J, Yang Y, Wei W, et al. Graphgpt: Graph instruction tuning for large language models[J]. arXiv preprint arXiv:2310.13023, 2023.

Reviewer comment (Theory):
Is partial observability really a salient property of this specific set of 28 actions?

Author response:
We appreciate your inquiry regarding the salience of partial observability for action set in our study. Indeed, partial observability plays a crucial role in successfully executing these actions within the VirtualHome environment. To illustrate: (i) Manipulation-Related Actions: Actions such as 'Grab' and 'Touch' are successful only when the robot is close to the target object. This proximity is a crucial aspect of the robot's partial observation, determining the feasibility of these actions. (ii) Navigation-Related Actions: Actions like 'Walk' and 'Find' are available only when the robot and the target object are in the same room. (iii) Object Accessibility: The robot's ability to interact with objects is also governed by partial observability. For example, if an object is placed inside a closed container, it becomes invisible to the robot's observation, rendering the object inaccessible for any interaction.

These examples underscore that partial observability is not just a theoretical consideration but a practical constraint that significantly influences the behaviors of the robot.

Reviewer comment (Experiments):
While the method clearly shows good performance on the considered domains, the experiments could be improved to address important questions about the approach, namely prompt engineering and the most efficient use of an LLM in task planning.

Author response:
Thank you for pointing out the potential improvement of our methodology. We have included additional experiments on varying prompting strategies. The corresponding result is presented in response to **Q2.1**. The experiment result shows that different prompts yield varying impacts, yet the prompts we currently employ maintain a sufficiently high level of effectiveness compared to others.

Reviewer comment (Experiments):
How good is the current prompt used in the plan sampling step? Given that errors made here are currently catastrophic and the prompt needs to be task-specific, a careful analysis of the quality of this prompt would be interesting.

Author response:
Thanks for your insightful feedback! In response to your query about the sensitivity of plan sampling's performance to the prompt used, we have conducted additional experiments to understand this aspect. The original prompts in our paper were designed following the guidelines discussed in [1], and we explored three variants of the prompting strategy:

(i) GPT-4 Optimized Prompt: This variant involves using GPT-4 to optimize our original prompt based on [Prompt Engineering Guide from OpenAI](https://platform.openai.com/docs/guides/prompt-engineering).

(ii) Code-Style Prompt[2][3]: Transforming the original prompt into a Python code format. In this variant, the prompt generates code and produces comments that serve as "thoughts," aiding in the clarity and interpretability of the plan sampling process.

(iii) Chain-of-thought[4]: Tasking LLMs to first generate a comprehensive chain of thought before sampling possible plans.
    
These experiments were conducted under the condition of N=50. As is shown in the table, GPT-4 Optimized Prompt and Code-Style Prompt does not lead to performance improvement. In contrast, using a chain of thought prompt enhances performance (+0.44 success rate).

The negative impact on performance observed when using code-style prompts may be attributed to the limited code-generation ability of GPT3.5. We will explore models specifically designed for code generation with code-style prompts in the future[5][6].

|                      | **EXEC ↑** | **SR ↑** | **GCR ↑** | **Cost ↓** | **No.Correction ↓** |
|----------------------|------------|----------|-----------|------------|---------------------|
| TreePlanner       | 88.26±2.47 | 41.58±3.20| 59.55±3.20 | **4.54**±0.16  | **2.04**±0.26           |
| GPT-4 Optimized Prompt | **89.52**±2.15 | 39.25±0.54 | 58.80±1.05 | 4.73±0.13 | 2.06±0.18          |
| Code-Style Prompt    | 70.70±3.24 | 37.10±0.89 | 50.74±2.73 | 6.25±0.57 | 3.12±0.31          |
| Chain-of-thought    | 89.13±2.04 | **42.02**±1.27 | **60.13**±1.89 | 5.72±0.91| 2.38±0.17          |

[Reference]
    
[1] Vemprala S, Bonatti R, Bucker A, et al. Chatgpt for robotics: Design principles and model abilities[J]. Microsoft Auton. Syst. Robot. Res, 2023, 2: 20.

[2] Singh I, Blukis V, Mousavian A, et al. Progprompt: Generating situated robot task plans using large language models[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 11523-11530.
    
[3] Liang J, Huang W, Xia F, et al. Code as policies: Language model programs for embodied control[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 9493-9500.

[4] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.

[5] Xu Y, Su H, Xing C, et al. Lemur: Harmonizing Natural Language and Code for Language Agents[J]. arXiv preprint arXiv:2310.06830, 2023.

[6] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.

Reviewer comment (Experiments):
Another open question is whether it's more efficient to restrict the use of the LLM to plan sampling alone. See the questions for more details. (I'd be open to adjusting my score based on the responses to these questions.)

Author response:
Thanks for your suggestion. Corresponding experimental results and discussion are presented in the response to **Q4** and **Appendix C**.

Reviewer comment (Experiments):
The set of tasks considered are from a single domain (VirtualHome). This makes it difficult to evaluate the broader utility of the proposed ideas.

Author response:
Thank you for your insightful suggestion regarding exploring additional tasks and environments beyond VirtualHome (VH) in our research. In response to your query, we are open to and considering including new tasks and environments in our future work.

To clarify, our initial choice of VH was due to its complexity and real-world-like scenarios(refer to Appendix A and Appendix F). This complexity underscores the challenge of token inefficiency, a central focus of our study. However, we recognize the importance of diversifying our experiments to comprehensively assess our approach's applicability and robustness.

In contrast to VH, another popular embodied planning environment is Alfworld [1], which faces fewer challenges related to token inefficiency due to its simpler environment (single room, fewer objects). Regarding the specific mention of Toolbench, we find this suggestion intriguing. Toolbench, or similar environments, could offer unique challenges and scenarios that differ from those in VH. Their inclusion could enrich our understanding of token inefficiency in various contexts and further validate the effectiveness of our proposed solutions.

In the subsequent version of our paper, we will also include discussions on a broader range of planning environments[2][3][4]. We appreciate your valuable feedback, which has guided us to consider these important extensions to our work.

[Reference]

[1] Shridhar M, Yuan X, Côté M A, et al. Alfworld: Aligning text and embodied environments for interactive learning[J]. arXiv preprint arXiv:2010.03768, 2020.

[2] (Jericho) Hausknecht, Matthew, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. "Interactive Fiction Games: A Colossal Adventure." arXiv:1909.05398 [Cs], February 25, 2020. http://arxiv.org/abs/1909.05398.

[3] Liu X, Yu H, Zhang H, et al. Agentbench: Evaluating llms as agents[J]. arXiv preprint arXiv:2308.03688, 2023.

Reviewer comment (Reproducibility):
Some of the implementation details of the prompts, especially the one used in grounded deciding, are not fully described. See the questions for more details.

Author response:
Thanks for the opportunity to include more details about our research. Corresponding details are included in the response to **Q1**.

Reviewer comment (Reproducibility):
I was unable to identify what error information is included in the history of the prompt used for grounded deciding. Per Figure 2 and Appendix F.3, it seems like some information about previously failed actions seems to be included in the history part of the prompt. If yes, are failed actions restricted to those on the current trajectory from the root to the current node or are they a global history over the entire tree? Which variant is better and why?

Author response:
Thank you for the opportunity to provide more details. Error information typically refers to situations where an action cannot be executed due to its precondition not being met (e.g., unable to operate on objects inside a fridge before opening it with [Open] <fridge>). The error information included in the prompt used for grounded deciding is in the format of [trajectory 1], [error information 1], [trajectory 2], [error information 2], and so on, indicating a global history over the entire tree.

We chose this variant because we believe LLMs can optimize themselves implicitly with multiple failed trajectories. This is consistent with the viewpoint presented in [1]. LLMs can use error information as a feedback signal to improve their decision-making process and avoid repeating the same mistakes.

[Reference]

[1] Shinn N, Cassano F, Gopinath A, et al. Reflexion: Language agents with verbal reinforcement learning[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.

Reviewer comment (Experiments):
Would it be possible to include additional tasks beyond those in VirtualHome? For example, is Toolbench relevant here?

Author response:
Thank you for your insightful suggestion regarding exploring additional tasks and environments beyond VirtualHome (VH) in our research. In response to your query, we are open to and considering including new tasks and environments in our future work.

To clarify, our initial choice of VH was due to its complexity and real-world-like scenarios(refer to Appendix A and Appendix F). This complexity underscores the challenge of token inefficiency, a central focus of our study. However, we recognize the importance of diversifying our experiments to comprehensively assess our approach's applicability and robustness.

In contrast to VH, another popular embodied planning environment is Alfworld [1], which faces fewer challenges related to token inefficiency due to its simpler environment (single room, fewer objects). Regarding the specific mention of Toolbench, we find this suggestion intriguing. Toolbench, or similar environments, could offer unique challenges and scenarios that differ from those in VH. Their inclusion could enrich our understanding of token inefficiency in various contexts and further validate the effectiveness of our proposed solutions.

In the subsequent version of our paper, we will also include discussions on a broader range of planning environments[2][3][4]. We appreciate your valuable feedback, which has guided us to consider these important extensions to our work.

[Reference]

[1] Shridhar M, Yuan X, Côté M A, et al. Alfworld: Aligning text and embodied environments for interactive learning[J]. arXiv preprint arXiv:2010.03768, 2020.

[2] (Jericho) Hausknecht, Matthew, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. "Interactive Fiction Games: A Colossal Adventure." arXiv:1909.05398 [Cs], February 25, 2020. http://arxiv.org/abs/1909.05398.

[3] Liu X, Yu H, Zhang H, et al. Agentbench: Evaluating llms as agents[J]. arXiv preprint arXiv:2308.03688, 2023.

Reviewer comment (Experiments):
Given the paper's emphasis is on efficient use of the LLM, what performance improvement does the LLM offer in grounded deciding over other action selection mechanisms? For example, is it feasible to replace the use of the LLM in grounded deciding with "classical" action selection techniques (e.g., UCB, best-first heuristic search)? How might this perform? More generally, please discuss where exactly the LLM is required / beneficial over classical search-based planners.

Author response:
We appreciate your insightful question regarding using LLMs in grounded deciding compared to classical action selection techniques. Grounded deciding in our approach essentially represents a best-first search that employs Large Language Models (LLMs) as an implicit, pre-parametrized evaluation function.

Due to the necessity of conducting extensive simulations within the environment, Monte Carlo Tree Search (MCTS) frequently encounters inefficiency issues, resulting in an unjust comparison with alternative methodologies that do not require such extensive simulation.

    
We further compared grounded deciding (LLM-based evaluation function) with other heuristic methods: **(i)** Language-Guided BFS: This method evaluates actions based on how likely they are to occur, as determined by a language model. It uses the "log probability" concept to estimate each action's likelihood. The higher the probability, the more weight is given to that action; **(ii)** Environment-Guided BFS: This method evaluates the action based on the observability of objects involved. Actions involving easily observable objects are given higher weight; **(iii)** Hybrid BFS: This method uses a parameter, α, to balance the two methods. If α is closer to 1, the language model's influence is stronger; if it is closer to 0, the environmental observability takes precedence. More details and formalization of the three heuristic methods can be found in the **Appendix C.5** in the updated version of paper.

    
|                  | **EXEC ↑** | **SR ↑** | **GCR ↑** | **Cost ↓** |
|------------------|------------|----------|-----------|------------|
| $TreePlanner_{N=50}$   | 49.01±5.67 | 28.14±2.45| 35.84±4.20 | 3.48±0.04  |
| $LanguageGuided_{N=50}$ | 34.71±0.65 | 15.06±1.02 | 22.47±1.72 | 2.16±0.02  |
| $EnvironmentGuided_{N=50}$ | 37.26±0.17 | 6.24±0.65 | 12.10±1.41 | 2.16±0.02  |
| $Hybrid_{\alpha=0.5}$ | 38.71±0.42 | 17.74±0.69 | 25.34±0.95 | 2.16±0.02  |

As demonstrated by the experimental results, although heuristic methods exhibit a greater advantage in terms of token efficiency, their SR falls short compared to grounded deciding (using LLM as an implicit evaluation function). Moreover, combining both evaluation functions yields even greater performance improvements, thus demonstrating that there is still room for enhancement through optimizing the evaluation function in heuristic methods.


    
In summary, this comparison showcases the unique advantage of grounded deciding: (i) LLM's ability to implicitly construct a robust evaluation function that adapts to various situations without the need for extensive simulations; (ii) While classical heuristic searches are feasible, the process of constructing an evaluation function is often time-consuming and laborious.

Reviewer comment (Theory):
Not clear if the policy is grounded in the environment or in the LLM implicit distribution. Section 3.1 mentions the initial observation as part of the prompt, but the prompts in section F include a long “(observation)” for iterative planning and the grounded-deciding phase, but not for plan sampling. If a detailed initial observation is given, then it’s possible that many of the actions are grounded on them, reducing the significance of seeing the algorithm as dealing with a POMDP. If a detailed initial observation is not given, then the LLM might be retrieving possible courses of action given the initial description but not grounded in the environment. Both scenarios diminish the apparent significance of the work.

Author response:
Thank you for highlighting this aspect. In our experiments, detailed initial observations were included in the prompt. About "*the LLM potentially retrieving actions based on the initial description rather than being environmentally grounded*": Firstly, while the LLM's initial plan sampling might not be directly grounded, it provides a broad search space. In the grounded deciding stage, the LLM can select environmentally grounded actions from this set. Secondly, the LLM is prompted to generate plans conforming to commonsense logic during the plan sampling stage, as indicated in "*The temporal relationship between subtask sequences must adhere to commonsense logic.*" (Appendix F). This approach ensures that the LLM considers dynamic environmental changes, reducing reliance on the initial state alone and enhancing the practical relevance of our model in dealing with POMDP scenarios. We hope these perspectives will be considered, and we sincerely appreciate the opportunity to clarify these points.

Reviewer comment (Experiments):
No discussion of the systematicity of LLM-based planning. LLM-based planning is a popular topic, but the increasing body of work might call for a more careful examination. Each paper in this direction is an opportunity to examine the problem and the methodology. In general, planning seems easier in the “happy path,” where text with high likelihood matches what might work in an environment, but things can get more complicated quickly. For example, the task of “calling Mary using the cell phone” requires exploration beyond assumptions about the phone’s location. While the prompt in the grounded deciding phase includes the full observation, some crucial clues may be present but not effectively used in the plan sampling phase. It’s possible that finding an object like a cell phone might require samples beyond a fixed N, and no alternative is offered. The key underlying problem is that a flexible plan dealing with observations cannot be found using a fixed number of samples; some tasks might require fewer samples and some more, which is why planning algorithms use search.

Author response:
We appreciate your insightful observations regarding the systematicity and flexibility of LLM-based planning. Your concern about the fixed number of samples (N) in planning tasks is well-founded and highlights a crucial aspect of our research.

We have conducted additional experiments using GPT-4 to dynamically adjust the sampling number (N) based on the specific task. This approach allows the model to propose a more tailored sampling N and plan length, depending on the task's complexity and context.

As detailed in our response to **Q11.1**, these experiments revealed that while the model's prediction of the number of steps has a reasonable mean error (**3.19** steps), there is significant variability in performance across different tasks, as indicated by a standard deviation of **2.73** steps. This variability underscores the challenge of predicting task complexity with high accuracy.

Moreover, we observed a moderate positive correlation between task complexity (measured by the number of steps) and the suggested Sampling_N. This finding aligns with the intuitive notion that more complex tasks require more samples for exploring a more comprehensive range of potential plans.

In conclusion, dynamically adjusting the sampling N based on the task is a promising method to address the concern you raised. However, it is clear that the LLM's capability to predict the number of steps accurately still requires further improvement. This could be achieved through advanced techniques such as prompt engineering and in-context learning, which we plan to explore in our future work. Your feedback has been invaluable in guiding these efforts and ensuring our methodology is robust and adaptable to various planning scenarios.

Reviewer comment (Experiments):
Experimental significance issue due to a single environment with only 35 tasks. The dataset subsection explains that 35 tasks were used, but the tasks were not listed. The VH has many other tasks. The 35 tasks may be biased in a way that affects the significance of the observations.

Author response:
Thank you for pointing this out. We have tried to avoid introducing bias in the process of dataset construction.

In earlier planning papers based on VirtualHome, evaluations were based on the Longest Common Subsequence (LCS) and human evaluations [1]. While this applies to the entire VirtualHome dataset, it has two main drawbacks: 
1) The LCS metric struggles to effectively assess the actual impact of a plan in an environment; 
2) Human evaluation is highly labor-intensive.

Therefore, we referred to the method of using success rate to evaluate the success of a plan [2]. However, some tasks in VirtualHome are only suitable for some types of automated testing. For example:

**Example 1:**
Task Name: *Juggling*
Description: The robot would catch any items thrown at it and immediately start to juggle them at a pro level. It would have a max of 6 things it can juggle at once. That would be awesome to watch.
Explanation: The task name is ambiguous, and the task's goal conditions are unclear.

**Example 2:**
Task Name: *Check Email*
Description: Turn on the laptop. Connect to the Internet. Type in the URL. Check for new email messages.
Explanation: As it is impossible to obtain information about "checking email" in VirtualHome, the final goal conditions might include On(computer), but it is impossible to determine if the robot checked the email, allowing for potential cheating by the planner.

The annotation process mainly involved filtering out VirtualHome parts unsuitable for automated testing (success rate). Thus, we believe this annotation method avoids introducing bias as much as possible, allows for better-automated testing, and avoids the subjectivity of human evaluation and the limitations of LCS.

[References]

[1] Huang W, Abbeel P, Pathak D, et al. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents[C]//International Conference on Machine Learning. PMLR, 2022: 9118-9147.

[2] Singh I, Blukis V, Mousavian A, et al. Progprompt: Generating situated robot task plans using large language models[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 11523-11530.

Reviewer comment (Experiments):
Experimental significance issue due to using only VH. There are other environments for testing these ideas. The related work section must cover what other alternatives are used in related work and why it’s a good idea to select only this environment. Example: (Jericho) Hausknecht, Matthew, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. “Interactive Fiction Games: A Colossal Adventure.” arXiv:1909.05398 [Cs], February 25, 2020. http://arxiv.org/abs/1909.05398.

Author response:
**Q1. [What are some other environments where this method could be tested?]**
    
Thank you for your valuable suggestion. In the revised version of our paper, we will include a comparative analysis of the VH environment against other alternatives, particularly addressing why we chose VH for our experiments.

To elaborate, VH is a highly complex environment encompassing multiple rooms and objects, with intricate inter-object relationships that closely mimic real-world settings(refer to Appendix F). This complexity not only makes the environment more challenging for planners but also highlights the issue of token inefficiency more prominently. The abundance of tokens required for representing observations and global information in VH starkly contrasts with environments like Jericho[2], as referenced. 
This distinction is crucial for our study's focus on addressing token inefficiency in complex, real-world scenarios. 

Furthermore, in contrast to VH, another popular embodied planning environment is Alfworld [1], which faces fewer challenges related to token inefficiency due to its simpler environment (single room, fewer objects). In the subsequent version of our paper, we will also include discussions on a broader range of planning environments, such as those mentioned in [3] and others.

We believe this clarification will enhance the understanding of our choice and its relevance to the research objectives.

[Reference]

[1] Shridhar M, Yuan X, Côté M A, et al. Alfworld: Aligning text and embodied environments for interactive learning[J]. arXiv preprint arXiv:2010.03768, 2020.

[2] (Jericho) Hausknecht, Matthew, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. "Interactive Fiction Games: A Colossal Adventure." arXiv:1909.05398 [Cs], February 25, 2020. http://arxiv.org/abs/1909.05398.

[3] Liu X, Yu H, Zhang H, et al. Agentbench: Evaluating llms as agents[J]. arXiv preprint arXiv:2308.03688, 2023.

Reviewer comment (Experiments):
In general, given the absence of task descriptions and their plans, it’s hard to know whether the results are significant. Even if the 35 tasks and their plans were provided, the results might be an artifact of this particular environment. It is possible that the current version of GPT 3.5 performs better at household tasks than at tasks in other environments like navigating a store.

Author response:
We appreciate the opportunity to provide additional details about the dataset. This is presented in the response to **Q11**.

Reviewer comment (Evaluation):
Goal Condition Recall might be a misleading metric. Different subgoals might be easier than others, and the structure of the subgoals is not known.

Author response:
**W6.1 [Secondary issues - Goal Condition Recall might be a misleading metric. Different subgoals might be easier than others]**

We appreciate your insights regarding using Goal Condition Recall (GCR) as a metric in our study. GCR was chosen as an intermediate indicator alongside Success Rate (SR) to provide a more granular assessment of the system's performance in task planning. While SR is an excellent measure of absolute performance, it can be overly strict, and GCR offers a way to evaluate performance nuances that SR might overlook. Moreover, the effectiveness of GCR is not only in assessing absolute performance but also in comparing relative performance against baseline models. In summary, while we recognize the potential limitations of GCR as a standalone metric, its combination with SR provides a comprehensive and nuanced understanding of the system's performance.

**W6.2 [Secondary issues - we don't know the structure of the subgoals]**

In Section 4.1, we introduced goal conditions, and here, we provide a more comprehensive explanation of the structure of subgoals. Subgoals primarily consist of the eventual state of the relationship between two objects and the final desired state of a specific thing. Either part may be absent, but at least one subgoal is mandatory. Taking the task "*Make Coffee*" as an example, it has two goal conditions: one is *inside(coffee, coffee_maker)*, and the other is *on(coffee_maker)*.

Reviewer comment (Theory):
Confusions in the theoretical emphasis on POMDPs. The hallmark of POMDPs, being a Markov decision process, is the Markov assumption: decisions can be made just by looking at the state without the history. However, the policy in section 2 includes the history of actions but perhaps not the observation history. The emphasis on POMDPs is overstated, as the VH has high visibility while other environments feature more partial observability (for instance, Jericho).

Author response:
**W7.1 [Secondary issues - Confusions in the theoretical emphasis on POMDPs. The hallmark of POMDPs, being a Markov decision process, is the Markov assumption: decisions can be made just by looking at the state without the history. However, the policy in section 2 includes the history of actions but perhaps not the observation history.]**

Thank you for raising your concern. We appreciate your interest in our approach and would like to take this opportunity to clarify how it aligns with the POMDP framework.

In traditional Markov Decision Processes (MDPs), the Markov assumption implies that decisions can be based solely on the current state without needing historical information. However, in the case of POMDPs, due to the inherent partial observability of states, it becomes necessary to consider historical data to estimate the current state better. This does not contradict the Markov assumption but instead adapts it to the context of partial observability. 

Moreover, our approach is consistent with methodologies applied in recent literature[1][2]. Our experiments were designed to ensure a fair comparison between the Tree Planner and other baseline models. We remain confident that including action history in our policy does not undermine the findings of our paper. We appreciate your consideration of these points.

[Reference]

[1] Singh I, Blukis V, Mousavian A, et al. Progprompt: Generating situated robot task plans using large language models[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 11523-11530.

[2] Li S, Puig X, Paxton C, et al. Pre-trained language models for interactive decision-making[J]. Advances in Neural Information Processing Systems, 2022, 35: 31199-31212.



**W7.2 [Secondary issues - Confusions in the theoretical emphasis on POMDPs. The focus on POMDPs is over-stated as the VH has high visibility while other environments feature more partial observability (for instance, Jericho).]**

We appreciate the opportunity to clarify this point further. Indeed, the partial observation in VirtualHome is primarily manifested in the visibility limitations within a room and the inability to observe objects inside closed containers. This setup provides more observability compared to environments like Jericho[1] or Alfworld[2], and we plan to include a comparison with these environments in our revised paper to highlight this difference.

However, we still maintain that VirtualHome satisfies the definition of partial observation. Our main objective is to distinguish between partial and complete observation in VirtualHome, as in full observation settings, task planning becomes simpler and does not require exploration of the environment. The partial observation in VirtualHome is more realistic for real-world scenarios. Moreover, while VirtualHome offers more observability than other partial observation environments, which may reduce the POMDP challenge to some extent, it also introduces unique issues related to token efficiency. This aspect adds value to our research and contributes to a more nuanced understanding of POMDPs in different settings.

[Reference]

[1] Shridhar M, Yuan X, Côté M A, et al. Alfworld: Aligning text and embodied environments for interactive learning[J]. arXiv preprint arXiv:2010.03768, 2020.

[2] (Jericho) Hausknecht, Matthew, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. "Interactive Fiction Games: A Colossal Adventure." arXiv:1909.05398 [Cs], February 25, 2020. http://arxiv.org/abs/1909.05398.

Reviewer comment (Experiments):
Global replan is non-comparable with other algorithms. Whether it’s possible to teleport to the initial state or not fundamentally changes the problem.

Author response:
**W8. [Secondary issues - Global replan is non-comparable with other algorithms]**

Thank you for your valuable feedback. The design of our global replan baseline model was primarily inspired by error correction approaches detailed in [1]. In specific scenarios, such as web interaction [2], the assumptions of global replan are particularly applicable. This can lead to an uneven comparison to some extent. In the next version of our paper, we plan to elaborate on this point, providing a more apparent distinction between global replan and other methodologies.

[Reference]

[1] Shinn N, Cassano F, Gopinath A, et al. Reflexion: Language agents with verbal reinforcement learning[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.

[2] Yao S, Chen H, Yang J, et al. Webshop: Towards scalable real-world web interaction with grounded language agents[J]. Advances in Neural Information Processing Systems, 2022, 35: 20744-20757.

Reviewer comment (Experiments):
The notion of inverse actions is a form of symbolic knowledge, leaving the scope of the paper. A more purist approach would task the LLM to undo the last k actions, perhaps offering the list of actions.

Author response:
We appreciate your insights regarding the concept of inverse actions in our paper. In response to Q7, we provided more details of inverse actions used in our study and explored the feasibility of learning inverse actions in real-world scenarios. 

We agree that employing a Language Model like GPT-4 to undo actions is viable. To this end, we conducted experiments where we provided GPT-4 with instructions (similar to that in Appendix F), a list of k executed actions, and their corresponding observations (formatted as $o_{0}$, $a_{1}$, $o_{1}$, ...). As demonstrated in the provided case study, the results show that GPT-4 can accurately generate the corresponding inverse actions.

*Case study:*

Task: Make coffee

Executed Actions:
- [Walk] <dining_room> (1)
- [Find] <coffee> (1)
- [Grab] <coffee> (1)
- [Find] <coffee_maker> (1)
- [Open] <coffee_maker> (1)
- [PutBack] <coffee> (1) <coffee_maker> (1)
- [Close] <coffee_maker> (1)
- [SwitchOn] <coffee_maker> (1)

Generated Undo Actions by GPT4:
- [SwitchOff] <coffee_maker> (1)
- [Open] <coffee_maker> (1)
- [Grab] <coffee> (1)
- [Close] <coffee_maker> (1)
- [Find] <_table_> (1)
- [PutBack] <coffee> (1) <_table_> (1)
- [Walk] <home_office> (1)

Reviewer comment (Evaluation):
The best GCR in section 5.2 is interesting, but GCR is not a good predictor of SR (success rate). In Table 2, the gap between GCR and SR is higher for Tree-Planner than for other approaches. Relatedly, Table 3 reports 45.5% of errors due to missing correct plan.

Author response:
**W10. [Minor - The best GCR in section 5.2 is interesting, but GCR is not a good predictor of SR (success rate).]**

See in response to Q6. We also update the corresponding SR results in Appendix C.4 in the updated version of the paper. Thank you for your insightful question regarding the relationship between GCR and SR in our experimental results. We appreciate the opportunity to address this critical point.

Reviewer comment (Novelty):
Algorithm 1 doesn’t add much. Any algorithm for building a “trie” data structure would work.

Author response:
We agree that other standard tree construction algorithms could serve as substitutes. The intention behind presenting Algorithm 1 was to delineate the process of building the action tree in TREE-PLANNER in a clear, algorithmic format. We recognize, however, that this has occupied a notable portion of the manuscript. To address this, in the next version of our paper, we plan to describe the action tree construction process in a textual format for brevity and move Algorithm 1 to the Appendix. This change will conserve space in the main text while still providing detailed methodological insights for interested readers.

Reviewer comment (Evaluation):
The paper should distinguish the world model from planning. Discussing the percentage of correct plans is useful but not actionable in a new domain as the set of possible plans in a domain is heavily unbounded.

Author response:
We thank the reviewer for their valuable feedback. In the next version of our paper, we will explicitly add content to distinguish between the world model and planning. We understand and acknowledge that calculating the proportion of correct plans becomes exceedingly difficult when the number of potential plans is unbounded. The plan sampling process effectively narrows down the space of possible plans to a more manageable scope. Despite the infinite possibilities of plans in each new domain, our method, through the use of LLM, intelligently filters out the most potential and practical plans.

Reviewer comment (Evaluation):
The directions suggested in error analysis (re-sample, chain-of-thought, self-reflection) might increase the context size, reducing one of the advantages.

Author response:
Yes, our discussion here primarily focuses on improving system performance. We agree that introducing these techniques requires considering the tradeoff between performance and efficiency.