LLM -based Open Domain Planning by Leveraging Entity-Attribute-Level Domain Models (2025)

Abstract: Currently, large language models (LLMs) based Open domain Natural language planning (LONG) has considerable room for improvement. E.g., non-reusable plans with incomplete intermediate states and missing steps hinder real-world applications. To remedy these flaws, this paper establishes a dataset with a baseline for LONG. The GOLD dataset provides the largest dataset for textual procedures, along with corresponding reusable formal planning domain definitions, to date. The baseline, DIGGER, leverages entity-attribute-level action models, which reveal relevant implicit physical properties (aka attributes) of salient entities in actions. DIGGER first extracts action models and builds typed entity lists from textual procedures. Then, it builds goal states for new tasks and instantiates grounded actions using domain prediction. At last, plans are generalized and translated into textual procedures by LLM. Reference-based metrics, LLM-as-a-Judge, and human evaluation are employed to comprehensively evaluate LONG. Experiments on GOLD validate that DIGGER is stronger and more generalizable than recently proposed approaches and LLMs. I.e., DIGGER is the best in seen domains and applicable to unseen domains without adaptation. Specifically, the BLEU-1 score increased from 0.385 to 0.408 on seen domains and rose to 0.310 on unseen domains.

## introduction large language models (llms) based open domain natural language planning (long 2 ) is ideal for real-world open domain planning @xcite , which uses natural language (nl). however, the text procedures drafted by humans or long contain flaws, such as incomplete states and missing steps @xcite . e.g., fig. 1 is a textual procedure on wikihow 3 for "how to graft a tree". there are at least three flaws in this plan. first, it ignores objects' positions (e.g., "hole"). second, details of "look after" are missing (e.g., "cut away any excess rootstock"). third, steps for preparing or waiting are absent (e.g., "wait for the graft to heal"). an appealing approach to boost the performance of long @xcite is to define domains in the planning domain description language (pddl) @xcite . however, there are three challenges in facilitating pddl in long: the scarcity of annotated data, the difficulty of building a complete domain model, and the challenges in nl-pddl translation. thus, to benchmark long, we propose dig-ger (llm-based open domain planning by leveraging entity-attribute-level domain models) and a dataset gold (natural language procedural texts with pddl definitions). gold includes 103 domains, 1100 tasks, and 1086 actions from four source datasets: wik-ihow @xcite , alfred-l @xcite , proc2pddl @xcite , and openpi2.0 @xcite . openpi, with human-drafted and planner-verified pddl definitions, is referred to as openpi+ in this paper. digger has two stages: training and planning. training with gold, digger builds domain clusters, entity-attribute-level action models, and typed entity @xcite lists. domain clustering prepares for domain prediction. then, with nl sentences and pddl definition as labels, a fine-tuned comet @xcite extracts action models. we find entity attributes (i.e., physical properties of entities) using conceptnet @xcite , and action instantiating relies on typed entity lists. planning via digger has six steps: 1) predicting the domain; 2) extracting goal states from nl using comet; 3) instantiating grounded actions @xcite with typed entity lists; 4) providing a few-shot prompt for llms; 5) planning via llms in pddl format; 6) generating text procedures. while evaluating long is challenging, we employ a comprehensive evaluation using multiple metrics. as previous studies, we uses referencebased metrics @xcite including bleu @xcite , rouge @xcite , meteor @xcite , and bertscore @xcite . we further use llm-as-a-judge (laj) and human evaluation in this paper. experiments on gold show that dig-ger outperforms three llms and two recently proposed approaches. the three llms are deepseek-v3 @xcite , gpt-4, and gpt-4o, and the two approaches are open-ground-planning @xcite (or ogp for short) and plan @xcite . specifically, in seen domains where ogp and plan can be applied without adaptation, the best bleu-1 score increased from 0.385 to 0.408. on unseen domains, while the best llm's bleu-1 score is 0.089 (gpt-4), digger's score is 0.310. our contribution can be summarized as follows: ## large language models based open domain natural language planning long aims to generate a textual procedure t p for a given nl task title tsk (e.g. "how to graft a tree?"). tsk is an nl goal g in terms of a desired state that involves objects (e.g., "a grafted tree"). t p should be an ex-ecutive series of (maybe unseen) grounded actions that lead to g. i.e., t p consists of a series of nl steps n l(a) ∈ n l(a), where n l(a) is an nl action (a short sentence) and n l(a) is the set of all possible actions. suppose llm (•) is the function of llms, a probabilistic predictor of language tokens. i.e., given a sequence of tokens @xmath0, ..., t n ), @xmath1, llms (as a function llm (•)) will output a new (reusable) t n+1 and associated probabilities p ⊆ c × r. see eq. 1. the goal of long is to acquire t p for tsk. we first generate a pddl plan π for tsk with a partial domain d. then, we prompt llms to generate t p . i.e., t p ← llm (π) ← llm (d, tsk). ## conclusion while there are many ways to utilize ai planning, this paper will hopefully spark further developments in long. first, considering that a high-quality pddl definition is costly and unavailable under most circumstances, digger leverages reusable, incomplete domain models with entity-attribute-level information to identify the hidden, fine-grained state information in intermediate states. second, benefit from the fact that llms can tolerate nl-pddl translation-induced errors; extra validation mechanisms via (incomplete) pddl definitions boost the performance of long. third, as the most extensive dataset for textual procedures with corresponding pddl definitions, gold is indispensable for long.