PDDL P LANNING WITH P RETRAINED L ARGE L ANGUAGE M ODELS
A BSTRACT
We study few-shot prompting of pretrained large language models (LLMs) towards solving PDDL planning problems.
We are interested in two questions: (1)
To what extent can LLMs solve PDDL planning problems on their own?
(2) How and to what extent can LLMs be used to guide AI planners?
Recent work by
Valmeekam et al.
(2022) presents negative evidence for (1) in the classic blocks world domain.
We confirm this finding, but expand the inquiry to 18 domains and find more mixed results with a few clear successes.
For (2), we propose a simple mechanism for using good-but-imperfect LLM outputs to aid a heuristic-search planner.
We also find that the LLM performance is due not only to syntactic pattern matching, but also to its commonsense understanding of English terms that appear in the PDDL.
Code: https://tinyurl.com/llm4pddl
I NTRODUCTION
We are interested in the applicability of pretrained large language models (LLMs) (Brown et al., 2020; Chen et al., 2021; Chowdhery et al., 2022) to AI planning problems.
Research in AI planning over the last two decades has led to powerful domain-independent planners (Helmert, 2006) and a wide variety of real-world applications (Sohrabi, 2019).
AI planning problems are typically represented in PDDL (Fox & Long, 2003), a Lisp-like specification language.
Given an initial state, goal, and (partial) transition model written in PDDL, AI planners are capable of generating actions that achieve the goal from the initial state.
Do
LLMs have similar capabilities?
Can the commonsense understanding of an LLM be used to enhance the syntactic reasoning of an AI planner?
Towards answering these questions, we consider a few-shot prompting paradigm (Brown et al., 2020) where an LLM is queried with N examples of PDDL problems and solutions and 1 new problem from the same domain (Figure 1).
The hope is that the LLM will complete this prompt with a valid solution to the new problem.
There are several reasons to immediately doubt the viability of this enterprise.
First, since LLMs are pretrained on natural language or general code, there is no guarantee that their outputs will even be syntactically valid PDDL.
This lack of specificity to PDDL is a limitation that other learning-to-plan approaches do not have (Yoon et al., 2008; Jim´enez et al., 2019; Karia & Srivastava, 2021; Gehring et al., 2022).
Second, unlike planning-based approaches, the LLM does not have direct access to the transition model (PDDL operators), since it is prompted only with initial states, goals, and solutions 1 .
Third, and perhaps most worryingly, recent work by Valmeekam et al.
(2022) presents compelling evidence that “large language models still can’t plan” in the classic blocks world PDDL domain.
Despite these grounds for caution, we are motivated to continue for a few reasons.
First, pretrainined
LLMs have shown remarkable reasoning and problem-solving abilities in other contexts (Wei et al., 2022;
Drori et al., 2022; Lewkowycz et al., 2022).
Second, even though LLMs do not have direct transition model access, the example solutions may contain enough information for the LLM to generalize to new problems.
For example, PDDL typically contains English names that a human, and perhaps the LLM, could use to understand the domain.
The solutions may also exhibit a clear pattern that the LLM could potentially extrapolate, as in few-shot imitation learning (Cropper, 2019;
Silver et al., 2020).
Third, even if LLMs “still can’t plan” on their own, they may still provide useful guidance to an AI planner.
In this work, we apply OpenAI’s Codex LLM (Chen et al., 2021) to 18 PDDL domains: 17 from the
Pyperplan benchmark collection (Alkhazraji et al., 2020) and 1 new PDDL domain.
We choose to go “wide” in this pilot study in an effort to identify domains that merit future “deep” investigations.
We find preliminary evidence to support the following claims:
In certain PDDL domains, LLMs are capable of solving some nontrivial problems.
However, in many other cases, LLMs cannot yet solve problems on their own.
LLMs are sensitive to the semantics of the English terms used in the PDDL problems.
LLMs can perform well in novel 2 PDDL domains.
Even when LLMs cannot solve problems, they can still be useful for planning.
Beyond our empirical findings, we make two additional contributions.
(1) We propose a simple but effective mechanism to leverage the output of an LLM during planning.
The main idea is to use the LLM output to initialize the queue of a greedy best-first search planner.
(2) We contribute 3 an open-source codebase that can be used to benchmark the planning performance of future LLMs, and to develop new techniques for combining LLMs and planners.
R ELATED W ORK
There has been considerable recent interest in leveraging LLMs for decision making (Collins et al., 2022;
Cohen et al., 2021; Simon & Muise, 2022).
For example, Sharma et al.
(2022) fine-tune a pretrained LLM to generate sequences of natural language instructions for robotic tasks in the
ALFRED household environment (Shridhar et al., 2020).
Li et al.
(2022) consider a similar finetuning strategy in the VirtualHome environment (Puig et al., 2018) and additionally propose an active data gathering process.
Huang et al.
(2022a) also consider VirtualHome, but with few-shot prompting for instruction generation.
They propose a mechanism for constraining the LLM output to feasible action sequences, which we adopt in this work (Section 4).
Huang et al.
(2022b) incorporate feedback into the prompts to recover from skill execution failures.
Ahn et al.
(2022) use LLM embeddings and pretrained skill value functions to map natural language instructions to skills in a large suite of robotics tasks.
Most relevant is recent work by Valmeekam et al.
(2022), who consider solving PDDL blocks world problems with LLM few-shot prompting and propose blocks world as a benchmark for improving decision-making with LLMs.
Their “plan generalization” setting is the most similar to ours.
One difference is that we prompt with the original PDDL syntax, rather than domain-specific natural language encodings of the PDDL.
Accordingly, we use Codex — a version of GPT-3 fine-tuned on general code — instead of GPT-3 itself.
4 Valmeekam et al.
(2022) come to a largely negative conclusion about LLM planning ability (“large language models still can’t plan”).
In looking beyond blocks world, we find evidence to support a more optimistic view.
We also show that LLMs can be useful to guide planning, even in domains where LLMs alone struggle to find complete plans on their own.
For this reason, we also evaluate on a new domain (“Dressed”) that we created after the LLMs were pretrained.
However, preliminary experiments suggest that performance with GPT-3 is similar.
B ACKGROUND
We begin with a brief review of PDDL planning and LLMs.
See Geffner & Bonet (2013) and Brown et al.
(2020) respectively for more detailed introductions.
3. PDDL P LANNING
We consider AI planning in a setting with discrete and fully-observable states, finite actions, deterministic transitions, and goal-based problems with no other extrinsic feedback.
In each state s ∈ S , the agent can select an action a ∈ A from the set of applicable actions
A ( s ) ⊆
A .
A known transition model F : S ×A → S determines the next state after an action is selected.
The transition model is partial: F ( s, a ) is defined if and only if a ∈ A ( s ) .
A problem consists of an initial state s 0 ∈ S and set of goal states g ⊆ S .
A solution to a problem is a plan a =
( a 0 , . .
. , a n − 1 ) that results in a goal state, that is, s i +
=
F ( s i , a i ) for all 0 ≤ i < n and s n ∈ g .
A domain is characterized by a state space, an action space, a transition model, and a problem distribution.
In this work, our objective is to maximize the expected number of problems solved from a domain’s problem distribution within a wall-clock timeout.
We are thus concerned with satisficing, not optimal, planning.
We consider domains encoded in the STRIPS subset of PDDL (Fox & Long, 2003).
A state s is represented by a set of objects and a set of ground atoms .
Each ground atom (e.g., (on a b) ) consists of a predicate ( on ) and ordered object arguments ( a and b ).
The object set is constant for a problem but changes between problems.
Objects can also be typed (e.g., a - block ).
A problem goal g is also represented by a set of ground atoms and the goal is achieved in state s if g ⊆ s .
An action a is represented by a ground operator (e.g., (unstack a b) ), which consists of an operator ( unstack ) and ordered object arguments ( a and b ).
A ground operator a has preconditions — a set of ground atoms — and is applicable ( a ∈ A ( s )) if the preconditions are a subset of s .
The ground operator also has add effects and delete effects , each a set of ground atoms, which define the transition model: the add effects are added to the state, and the delete effects are removed from it.
PDDL specifies a Lisp-like syntax for each of these representations.
An AI planner consumes a PDDL domain and problem as input and produces a solution.
Most state-of-the-art (SOTA) planners (Helmert, 2006; Hoffmann, 2001) are variants of heuristic-based forward search, where the PDDL encodings are used to automatically derive a heuristic function.
In this work, we primarily use the Pyperplan planner with greedy best-first search (GBFS) and the hFF heuristic (Alkhazraji et al., 2020).
Pyperplan is not SOTA, but we prefer it in this work due to its clean implementation and ease of extensibility.
We also report results for the SOTA Fast Downward planner for comparison (Helmert, 2006).
3. L ARGE L ANGUAGE M ODELS (LLM S )
A large language model (LLM) is a model with a large number of parameters (on the order of 10 11 ) that is trained to represent a probability distribution over a large corpus of text (e.g. the webtext corpora (Gao et al., 2020)).
Though these models are trained simply to predict the likelihood of text conditioned on some previous context, they exhibit impressive performance on a variety of downstream tasks like 3-digit arithmetic and word problems (Lewkowycz et al., 2022).
An LLM takes a text query as input and predicts probabilities over following text.
We use only the most likely output 5 .
In this work, we use the Codex LLM
(Chen et al., 2021).
S OLVING PDDL P ROBLEMS WITH F EW -S HOT P ROMPTING
We now describe our approaches for using pretrained LLMs to solve PDDL problems.
All approaches use few-shot prompting, where N training problems and solutions are generated once offline per domain and prepended to each new evaluation problem from the domain that we wish to solve.
The LLM is then queried with this complete prompt and the resulting output is used to construct a plan in one of several ways.
We detail each of these steps in the following subsections.
4. P ROMPT C ONSTRUCTION
To create examples for few-shot prompting, we reserve N prompting problems from the domain distribution and use a planner 6 to generate one solution per problem.
In practice, we select the smallest 7 N problems for prompting from a benchmark set.
Following previous work (e.g., (Kojima et al., 2022)), we construct a prompt prefix by sequencing together Q: <problem> and
A: <solution> for each of the N examples.
The problem and solution are encoded in PDDL syntax.
For each new evaluation problem in the domain, we independently create a prompt by concatenating the prefix and Q: <new problem> A: .
See Figure 1 for a simple example.
4. LLM O UTPUTS TO PDDL P LANS
Given a prompt, we query the LLM and record its response until Q: is encountered (the stop token) or a maximum length is exceeded (the context window).
We then attempt to parse the response into a PDDL plan.
Our parser is relatively naive and assumes that the response is a sequence of actions in the form (<operator> <object> <object> . . . ) .
Since the LLM is unconstrained, it is certainly possible for the response to contain syntax errors, e.g., invalid names or operators with invalid arguments.
It is also possible that the response represents a plan that violates the preconditions of the action at some step or does not reach the goal.
We consider two strategies for dealing with malformed LLM outputs: No Validation and Soft Validation.
With No Validation , we simply skip over any invalid action over during output parsing.
With Soft Validation , we use the technique of Huang et al.
(2022a) to constrain the outputs to be valid PDDL plans.
In this autoregressive prompting technique, rather than generating an entire plan all at once, we generate one action at a time.
After each closed-parentheses expression is output by the LLM, we check to see if it already represents a valid action.
If not, we find the valid action that is “closest” to the generated expression, where distance is defined via cosine similarity in a pretrained embedding space.
We use the Sentence-BERT embedding model ( paraphrase-MiniLM-L6-v2 ) from Hugging Face in experiments (Reimers & Gurevych, 2019).
After each valid action is obtained, we first check to see if the current plan is a solution.
Otherwise, the valid action is appended to the prompt and the LLM is re-queried.
4. U SING LLM O UTPUTS TO G UIDE A P LANNER
The above techniques are designed to address the question of whether LLMs can solve PDDL problems on their own.
We are also interested in the questions of how and to what extent LLMs can be used to guide a PDDL planner.
To this end, we propose a simple approach — LLM Plan Guidance — for using the LLM output with a search-based planner.
Recall that heuristic search algorithms maintain a priority queue of nodes, where each node contains a partial plan and a corresponding state sequence.
Given the LLM output, we first parse it into a plan a =
( a 0 , . .
. , a n − 1 ) by either the autoregressive or non-autoregressive technique described in Section 4.2.
We then use the initial state s 0 and the transition model F to generate the corresponding state sequence ( s 0 , . . .
, s n ) .
Then for each step i where 0 ≤ i < n , we add to the initialized priority queue a node with the partial plan ( a 0 , . .
. , a i ) and the corresponding state sequence ( s 0 , . . .
, s i +1 ) .
We use the standard priority function to order these nodes.
For example, in GBFS, the priority is equal to the heuristic.
To motivate LLM Plan Guidance, consider first the case where the LLM output already encodes a solution.
Assuming a well-behaved heuristic, GBFS will immediately pop the full plan a node from the priority queue and return the solution before expanding any other nodes.
Consider instead a case where the LLM outputs an almost complete plan, but it is missing one or a few actions at the end.
If the heuristic is a good approximation of the true cost-to-go, GBFS would again pop the full plan and continue searching from the end, likely exploring fewer nodes than it would starting from the initial state.
The LLM would provide especially useful guidance in the case where its output is sufficient for the planner to escape from some early local optima in the heuristic landscape.
The extent to which this LLM guidance benefits the planner is ultimately an empirical question; to answer it, we now turn to experiments.
E XPERIMENTS AND R ESULTS 5. D OMAINS
We use the Pyperplan domains and problems for our main experiments (Alkhazraji et al., 2020).
Pyperplan includes 21 domains.
We exclude 4 domains whose operators vary per problem (Airplane,
Openstack, Parcprinter, and PSR) and use the remaining 17 domains.
We sort the problems by length and use the first N = 2 problems for prompting and the next 10 problems for evaluation.
Since the Codex LLM was pretrained on open-source code (Chen et al., 2021), and since the Pyperplan benchmarks are open-source, it is possible that some or all of these domains appeared in the pretraining data 8 .
To mitigate the possibility that our results are due to memorization, we designed a new “Dressed” PDDL domain that, to the best of knowledge, is qualitatively dissimilar to existing
PDDL domains.
This domain involves planning for one or more people to get dressed for a casual or formal event.
Problems are procedurally generated and vary in the number of people, events, and clothing.
We again use N = 2 small train problems and 10 larger evaluation problems.
5. A PPROACHES
We now describe the approaches and baselines that we evaluate on all domains
. 5.2. O PEN -L OOP A PPROACHES
Our first set of “open-loop” approaches are designed to address the question of whether LLMs can plan on their own.
These approaches include:
• Soft Validation: Our main approach with autoregressive prompting as described in Section 4.2.
• No Validation: Same, but without autoregressive prompting.
• Ablate Names:
Same as No Validation, but each operator, object, type, and predicate name is randomly replaced with one of the 10,000 most common English words (Price, 2022).
• Random Actions: Selects a random applicable action for up to n steps or until the goal is reached, where n is equal to the length of the Soft Validation output per problem.
5.2. P LANNING - BASED A PPROACHES
Our second set of approaches are designed to address the question of whether LLMs can be useful in guiding a heuristic search planner.
We use Pyperplan’s GBFS search with the hFF heuristic unless otherwise noted with a 300 second planning timeout (LLM query time not included).
These approaches include:
• LLM Plan Guidance: Our main approach (with autoregressive prompting) where the LLM output is used to initialize the GBFS priority queue (Section 4.3).
• LLM Plan Guidance
No Val: Same, but without autoregressive prompting.
• Random Plan Guidance:
The random plans generated by the Random Action baseline from Section 5.2.1 are used to initialize the queue instead of the LLM outputs.
• Pure Planning: The Pyperplan planner with no additional guidance.
• Fast Downward: The Fast Downward planner with the lama-first configuration (Helmert, 2006).
For space, these results are deferred to the appendix.
5. E XPERIMENTAL S ETUP
All experimental results are reported over 5 random seeds.
Variation between seeds is driven by nondeterministism in Pyperplan’s implementation of hFF, random action selection in the baselines, random object renaming in the Ablate Names ablation, and small variations in processor speed.
All
LLM-based approaches use the code-davinci-002 Codex LLM (Chen et al., 2021).
Queries are made through the OpenAI API with temperature 0.0.
Experiments were run on Ubuntu 18.
using 4 CPU cores of an Intel Xeon Platinum 8260 processor.
5. R ESULTS AND A NALYSIS
[Table]
Table 1 reports the fraction of the 10 evaluation problems solved by each of the openloop (non-planning) approaches in each of the domains, averaged over 5 seeds.
Our first conclusion is that in certain PDDL domains, LLMs are capable of solving some nontrivial problems .
Figureshows one case where the LLM generalizes from a Gripper prompting problem with 4 balls to an evaluation problem with 6 balls.
This is the easiest case; the LLM also generalizes to all evaluation problems in the set, including the largest withballs and 77 required actions.
The Gripper domain is also the simplest in terms of the extrapolation required, since all problems can be solved by a sequence of repeating pick, pick, move, drop, drop actions.
Other domains like Miconic, Rovers, Scanalyzer, TPP, and Woodworking require considerably more extrapolation, and in each of these cases, the Soft Validation approach achieves nontrivial performance.
Our second conclusion is that in many other domains, LLMs cannot yet solve problems on their own .
This is evident from the zero entries in the Soft Validation column, including in the Blocks domain, which confirms the findings of prior work (Valmeekam et al., 2022).
Upon inspection, we find that gross syntax errors in the output of the LLM are very rare, but mistaken actions are common, and occur both in the operators and object arguments.
We also find that the LLM struggles with certain spatial concepts, e.g., understanding that a block tower needs to be built from the bottom up, or that an elevator should move up to reach higher floors and down to reach lower floors.
The difference in performance between No Validation and Soft Validation supports previous findings (Huang et al., 2022a) that constraining the LLM output to valid plans can lead to substantially better performance.
However, it is worth noting that the autoregressive prompting required by Soft
Validation dramatically increases the approach run time, since there is significant latency in querying the LLM.
We also conducted preliminary experiments with the dynamic prompting method proposed by Huang et al.
(2022a), but did not find significant improvements in our setting, likely due to a limited number of prompting problems from which we could select.
By comparing No Validation to Ablate Names, we see a sharp decline in performance, suggesting that LLMs are sensitive to the semantics of the English terms used in the PDDL problems.
If the LLMs were instead using syntactic pattern matching to generalize from the prompts, we would expect to see minimal performance decline when terms are substituted.
Additional experiments with random strings, rather than English words, supported the same conclusion.
The perfect evaluation performance of Soft Validation in the Dressed domain supports the claim that
LLMs can perform well in novel domains , not just in domains that may have existed in some form in the LLM pretraining corpus.
[Table]
Elevators
[Table]
Gripper
[Table]
Miconic -
[Table] [Table]
Movie -
[Table]
Pegsol
[Table] [Table]
Rovers -
[Table] [Table]
[Table] [Table]
[Table]
Table 2 reports three metrics for each of the planning-based approaches: the fraction of theevaluation problems solved, the average number of nodes created during planning, and the number of nodes expanded.
All metrics are reported as a percentage increase or decrease relative to the
Pure Planning approach.
For fair comparison, node creation and expansion results are averaged over only the problems solved by all planning-based approaches.
These results suggest that even when
LLMs cannot solve problems, they can still be useful for planning .
For example, in the Depot,
Elevators, Logistics, Satellite, and Sokoban domains, Soft Validation does not solve any problems alone, but LLM Plan Guidance strictly improves over Pure Planning.
By comparing LLM Plan
Guidance and Random Plan Guidance, we also confirm that the improved performance is not simply due to initializing the queue with arbitrary partial plans; the LLM is providing nontrivial guidance.
Limitations.
Although these results are encouraging, note that SOTA AI planning remains far more efficient than LLM Plan Guidance in terms of wall-clock time.
There is high latency in querying the LLM, and there is a marked gap between the Pyperplan planner and the SOTA Fast Downward planner (see Table 7).
Another limitation of current LLMs is their fixed-size context window for prompts and responses (4096 tokens for Codex).
This context window can be a major bottleneck for PDDL.
We experimented with prompt compression techniques, but found that using N >examples consistently degraded overall performance because of exceeded context windows.
We do see that N = 2 is better than N = 1 , which suggests that if the context window could be increased to accommodate larger N , results may improve further.
Despite these caveats, the results we present here are promising and motivate further research in using LLMs for PDDL planning.
D ISCUSSION AND C ONCLUSION
In this work, we investigated the extent to which LLMs can plan on their own and be used to guide planning in PDDL domains.
We found that LLMs alone can solve some nontrivial PDDL problems, but fail to solve many others.
However, even in problems that LLMs cannot solve, we showed that their outputs can be useful in guiding a heuristic-search planner.
There remains a large gap between
SOTA and LLM-based planners, but our results suggest that further research is well-motivated.
We hope that our open-source code can serve as a benchmark for progress in planning with LLMs and can help foster collaboration between the LLM and PDDL planning communities.
A CKNOWLEDGEMENTS
We thank Gabe Grand, Michael Katz, Wenlong Huang and the FMDM reviewers for helpful feedback on an earlier draft.
We gratefully acknowledge support from NSF grant 2214177; from AFOSR grant FA9550-22-1-0249; from ONR grant N00014-18-1-2847; from the MIT-IBM Watson Lab; and from the MIT Quest for Intelligence.
Tom and Nishanth are supported by NSF Graduate Research
Fellowships.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors
A A DDITIONAL R ESULTS
Here we report our full set of results with both means and standard deviations.
Table 3 includes all standard (non-planning) approaches.
Table 4 reports LLM Plan Guidance results.
Table 5 reports Random Plan Guidance results.
Table 6 reports Pure Planning results.
Table 7 reports Fast
Downward results.
Table 8 reports
LLM Plan Guidance
No Autoregress results.
[Table] [Table]
[Table] [Table]
[Table] [Table]