Open Grounded Planning: Challenges and Benchmark Construction (2024)

Abstract: The emergence of large language models (LLMs) has increasingly drawn attention to the use of LLMs for human-like planning. Existing work on LLM-based planning either focuses on leveraging the inherent language generation capabilities of LLMs to produce free-style plans, or employs reinforcement learning approaches to learn decision-making for a limited set of actions within restricted environments. However, both approaches exhibit significant discrepancies from the open and executable requirements in real-world planning. In this paper, we propose a new planning task–open grounded planning. The primary objective of open grounded planning is to ask the model to generate an executable plan based on a variable action set, thereby ensuring the executability of the produced plan. To this end, we establishes a benchmark for open grounded planning spanning a wide range of domains. Then we test current state-of-the-art LLMs along with five planning approaches, revealing that existing LLMs and methods still struggle to address the challenges posed by grounded planning in open domains. The outcomes of this paper define and establish a foundational dataset for open grounded planning, and shed light on the potential challenges and future directions of LLM-based planning.

## introduction human life is filled with tasks of varying complexities, from simple activities like brewing coffee to more substantive pursuits such as learning new skills. by utilizing our understanding of the world, we can formulate plans for tasks and execute these … workout every other day. ## related work large language models are trained on data containing extensive common knowledge and exhibit certain planning and common-sense reasoning abilities @xcite . prompting can be employed to guide large language models in generating plans for given tasks, and these plans often possess a high degree of freedom, making them challenging to execute in specific environments @xcite . to facilitate the 2 grounding of generated plans to an ai agent's executable action space, prior research has extensively explored grounded planning tasks (lin et al., 2023; wu et al., 2023). some approaches opt for a global planning strategy based on the task, aiming to directly generate plans that can be grounded to the execution environment in a single step (song et al., 2023; shen et al., 2023; yuan et al., 2023; wang et al., 2023a). conversely, other methodologies employ iterative interactive approaches as the primary means of plan generation to adapt to changes in the environment and conditions (ahn et al., 2022; wang et al., 2023b). however, these approaches often demonstrate limited effectiveness, completing constrained tasks with a finite set of actions within a singular domain. in open-domain environments, the enormity of tasks and action sets poses significant challenges, making it increasingly difficult to bridge the gap between plans generated by large language models and the execution of real-world tasks. therefore, in our work, we raise the challenge of open grounded planning and compile benchmark data from multiple domains ranging from everyday life to tool use and robot control scenarios, which consist of tens of thousands of tasks and actions. we also utilized our benchmark to assess the performance of mainstream proprietary models and open-source models with various planning methods on open grounded planning tasks. ## open grounded planning benchmark in this section, we initially present the task definition of open grounded planning and its associated challenges. subsequently, we introduce the open grounded planning benchmark, encompassing the construction of the dataset, evaluation metrics, and the methodology for automated assessment. ## methods in order to assess the performance of current mainstream models on the open grounded planning task, we endeavored to employ five distinct methods including retrieve and rewrite, a new framework we proposed, to address this challenge. task-retrieve: we first adopt a simple and intuitive approach by using the task name as the query for action retrieval. given a task t and an action set a, we retrieve the relevant action list a ′ = retrieve(t, a). llm selects and orders appropriate actions from this list to generate plan @xmath0, a ′ ). plan-retrieve: simply searching by task name makes it difficult to recall important steps that are not directly related to the task name. we try to specify the query to improve. we first force llm to generate an initial plan p 0 = {s 1 , s 2 , • • • , s n } for the task, then retrieve related actions based on the generated plan to get action list a ′ = retrieve(p 0 , a). finally, we perform similar selection and rearrangement as in task-retrieve. we also provide the initial plan for llm to refer to generate the final plan @xmath1, a ′ , p 0 ). ## experiment we systematically assess the capabilities of various llms and methods in the open grounded planning task by selecting and comparing mainstream proprietary models, open-source models, and opensource models fine-tuned with a small amount of domain-specific data. for each model, we examine the performance of different methods in section 4. we test the models' abilities in open grounded planning on both in-domain, the wikihow dataset, and out-of-domain, the tools and robot datasets. ## conclusion in this study, we introduced open grounded planning and developed a benchmark comprising datasets from various domains with vast action sets. extensive experiments revealed significant limitations in the performance of current models and methods in generating grounded plans for these sets. furthermore, we observed a pronounced challenge in enabling these models and methods to generalize from in-domain scenarios to out-of-domain datasets. compared to four other methods, our "retrieve and rewrite" approach demonstrates a partial resolution to the challenges inherent in open grounded planning. our work highlights the need for enhancing the capability of models and methods for expansive planning domains and improving the executability and quality of grounded plans, laying a foundation for future research.