PROC 2 PDDL : Open-Domain Planning Representations from Texts (2024)

Abstract: Planning in a text-based environment continues to be a significant challenge for AI systems. Recent approaches have utilized language models to predict planning domain definitions (e.g., PDDL) but have only been evaluated in closed-domain simulated environments. To address this, we present Proc2PDDL, the first dataset containing open-domain procedural texts paired with expert-annotated PDDL representations. Using this dataset, we evaluate the task of predicting domain actions (parameters, preconditions, and effects). We experiment with various large language models (LLMs) and prompting mechanisms, including a novel instruction inspired by the zone of proximal development (ZPD), which reconstructs the task as incremental basic skills. Our results demonstrate that Proc2PDDL is highly challenging for end-to-end LLMs, with GPT-3.5’s success rate close to 0% and GPT-4o’s 38%. With ZPD instructions, GPT-4o’s success rate increases to 45%, outperforming regular chain-of-thought prompting’s 34%. Our analysis systematically examines both syntactic and semantic errors, providing insights into the strengths and weaknesses of language models in generating domain-specific programs.

## introduction planning is the task of finding a sequence of actions to achieve a goal in a given environment @xcite . in real life, the environment is often described with natural language texts. to enable text-based, automated planning, recent work has used language models (lms) to generate plans @xcite . however, this approach is found to fall short with regard to both performance and interpretability @xcite . alternatively, another recent line of worked has instead used lms to translate the natural language description of environments to planning domain definition language (pddl) @xcite . this symbolic representation can then be solved by a planner in a plan @xcite @xcite @xcite . despite of the success of such a neurosymbolic method, all the above work has only been evaluated in closed-domains simulated environments such as a household (e.g., alfred @xcite )) or discrete object placement (e.g., blocksworld @xcite ) (as shown in to enable open-domain, text-based planning, we propose proc2pddl, a dataset to evaluate models' ability to generate pddl given procedural texts. proc2pddl consists of 27 pairs of open-domain procedures and pddl representations. each pddl representation include a domain file df that models the types, predicates, and actions, and a problem file pf that models the entities, initial states, and goal states, as illustrated in figure 1 . because proc2pddl is not bound 1 13 figure 2 : our formulation of the df action prediction task is as follows: given a natural language procedure text and a domain file header, a language model (lm) follows zone of proximal development (zpd) instructions in three sequential skills to predict domain actions, including parameters, preconditions, and effects. during evaluation, the predicted df is compared to a gold reference and used to solve corresponding pfs. to any simulation, the pddl representations are manually annotated by experts trained on this task to ensure validity, resulting in 27 domain files and 95 problem files. using this dataset, we study the task of action modeling @xcite formulated as follows. the input is some relevant natural language texts and the header of a df (i.e., types, predicates, and names of actions). based on a zpd instruction, the output is the domain actions in the df (i.e., parameters, preconditions, and effects). during evaluation, the predicted df is 1) compared to a ground-truth df as intrinsic evaluation, and 2) provided to a pddl solver with ground-truth pfs for the existence and correctness of plans as extrinsic evaluation. our system is delineated in figure 2 . in this formulation, our assumption of the df header is necessary to ensure the consistency of semantics between the df and the pf for evaluation. it is also empirically motivated; for example, a kitchen robot may have access to the types like 'ingredients' and predicates like 'diced' via some information extraction system given descriptive texts, but it may still need to predict, for "swinging a knife", the precondition that it is only safe to do so to the 'ingredients' and the effect that they will become 'diced'. through our experiment, we show that the task of action modeling in proc2pddl is highly challenging to state-of-the-art lms, where gpt-3.5 almost fails completely, gpt-4 can only generate exactly matching dfs 16% of the time and solvable pfs 33% of the time, and gpt-4o demonstrate 18% dfs accuracy and 37% pfs solving rate. by devising a zpd instruction that prompt lms to modularly generate pddl through extractioninference-translation approach, we improve action #df datasets ours 27 proc2pddl (wong et al., 2023) 2 minecraft, alfred (lyu et al., 2023) 1 saycan (xie et al., 2023) 2 blocksworld, alfred (liu et al., 2023) 7 blocksworld, etc. (huang et al., 2023) 1 tabletop (huang et al., 2022) 1 virtualhome (silver et al., 2022) 18 blocksworld, etc. (valmeekam et al., 2022) 2 blocksworld, logistics in our analysis, the syntactic errors indicate lms' weakness in generating low-resource and domainspecific programming languages @xcite like pddl, while the semantic errors suggest lms' inaccuracies to reason about actions and environments. ## task formulation the task of predicting a planning domain definition in a text-based environment can be seen as translating natural language texts to pddl symbolic language, which consists of a domain file (df) and one or more problem files (pfs). a df defines all actions in the environment: traditionally, the task of text-based pddl generation involves predicting pf based on text t, where a successfully generated pf can be solved by the predefined df. in this paper, we address an alternative formulation, action modeling (a), in which the generated df, given text t and the domain header h foot_0 , is capable of producing plans for pfs. ## dataset we introduce the proc2pddl dataset of 27 different t-df-pfs tuples, drawing procedural texts from wikihow articles of various topics (see appendix a). a class of graduate students in a u.s. university with prior knowledge of pddl are each given a wikihow article and annotate a df and multiple corresponding pfs from the article, each with a gold plan to solve it. on average, there are 13.33 defined actions in a df and 8.07 instantiated actions in a gold plan. in this work, all our data is used for evaluation, as all our methods are without task specific model training. some sample data of proc2pddl can be found in appendix b. ## methodology we first introduce a novel prompt design option, zpd, and then discuss the choices of text format (t), which can range from 10 to 2,000 tokens and influence the selection of lms. ## evaluation and analysis now that a model generates the parameters, preconditions, and effects for each action, we have a complete df. we evaluate it in two ways (figure 2 ). intrinsically, we semantically compare the predicted a with the ground-truth provided by our proc2pddl and report an action-wide accuracy. equivalence of two action definitions does not depend on the naming of variables nor on the order within conjunctions (detailed in appendix e). extrinsically, to measure actions' coherence, a bfsbased pddl solver foot_2 attempts to solve ground-truth pfs with the predicted df and a success rate is reported. an unsolved pf is caused by (1.) no plan can be found, or (2.) the solver runs for more than 30 seconds, or (3.) the solver returns an error (usually a syntax error in the generated pddl). the intrinsic and extrinsic results are shown in ## conclusion we present proc2pddl, the first open-domain dataset that juxtaposes natural language and planning domain definition language. our experiments show that zpd instructions facilitate lms' performance, while still find it challenging to translate the precondition and effects of actions. we hope our instruction design, evaluations and dataset help future progress towards integrating the best of lm and formal planning.