Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages (2025)

Abstract: Recent works have explored using language models for planning problems. One approach examines translating natural language descriptions of planning tasks into structured planning languages, such as the planning domain definition language (PDDL). Existing evaluation methods struggle to ensure semantic correctness and rely on simple or unrealistic datasets. To bridge this gap, we introduce Planetarium, a benchmark designed to evaluate language models’ ability to generate PDDL code from natural language descriptions of planning tasks. Planetarium features a novel PDDL equivalence algorithm that flexibly evaluates the correctness of generated PDDL against ground truth, along with a dataset of 145,918 text-to-PDDL pairs across 73 unique state combinations with varying levels of difficulty. Finally, we evaluate several API-access and open-weight language models that reveal this task’s complexity. For example, 96.1% of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, 94.4% are solvable, but only 24.8% are semantically correct, highlighting the need for a more rigorous benchmark for this problem.

## introduction recently, there has been growing interest in using large language models (llms) to solve planning problems. some research has focused on generating plans directly with llms @xcite @xcite @xcite . however, this approach has shown limited success; gpt-4 only achieves 35% accuracy on simple planning problems @xcite . another line of research uses llms to convert natural language prompts into structured planning languages, such as the planning domain definition language @xcite @xcite @xcite . early evidence suggests this method performs better than * these authors contributed equally to this work. correct goal block block pddl figure 1 : an example of one planning goal corresponding to many correct pddl goals. all pddl goals in the top row represent the displayed goal correctly. the bottom row illustrates pddl goals with different error types, showing instances that are solvable (a planner can generate a plan, but for a different planning problem), parseable (the pddl syntax is correct but will not produce any plan from a planner), and not parseable (it is not valid pddl). see section 5 for details. translating natural language to pddl enables a hybrid, best-of-both-worlds approach. the llm is responsible for interpreting natural language, and the resulting pddl can be given to traditional, symbolic planners that have been developed over decades @xcite @xcite @xcite . these traditional planners are efficient and ensure the correctness of their so-lutions. in contrast, leaving the planning to llms does not guarantee correctness, making them unreliable for critical applications. despite this shortcoming, llms are an enticing approach, as using traditional planners requires domain expertise and expertise in modeling planning problems. by using llms to translate natural language into pddl, we can better leverage the strengths of existing planners. using llms to translate natural language to pddl is an instance of a code generation task. evaluating code generation tasks, in general, is highly challenging. some benchmarks for code generation use match-based metrics that look for segments that overlap with ground truth code @xcite @xcite , but match-based metrics cannot cover the vast space of equivalent programs @xcite . therefore, benchmarks often test functional correctness using a suite of unit tests @xcite . for planning problems, existing work uses "plan validators" to check if the generated code can be solved with a traditional planner @xcite @xcite . we argue using validators alone is insufficient to determine if pddl generation is correct. this is because the llm can generate valid pddl that has nothing to do with the user's instructions and still be considered correct-a false positive. a rigorous evaluation of llms as generators of structured planning languages requires a precise definition of what it means for generated code to be correct. this is hard because many instances of pddl can represent the same planning problem, but it is not always obvious when they are equivalent (figure 1 ). properly checking pddl equivalence requires symbolic interpretation. to address this challenge, we introduce planetarium, a benchmark to evaluate llms on translating natural language descriptions of planning problems into pddl. our contributions are as follows: rigorous evaluation of pddl equivalence. we formally define planning problem equivalence and create an algorithm for checking whether two pddl problems satisfy this definition. this algorithm transforms the pddl code into scene graphs, computes an expansion of the goal states for both pddl problems, and then performs isomorphism checks between the graphs. our method ensures two pddl problems match if and only if they represent the same underlying planning task. we show how to make this algorithm efficient for three domains: blocks world, gripper, and a slightly simplified floor tile @xcite . ## related work llms as planners. there is growing interest in using llms to solve planning problems. several papers have demonstrated the potential of using llms for decision-making @xcite @xcite . some techniques that enable the use of llms as planners involve decomposing abstract tasks into concrete, atomic executable actions for an agent to perform @xcite . other approaches generate plan actions by scoring the possible next steps in a sequence of actions @xcite . when using llms to generate plans directly, the llm is given a natural language representation of a planning problem with the goal of generating a plan. we refer to this line of work as "llms as planners." one of the main findings is that llms have limited ability to generate and validate plans on their own, even for simple planning tasks @xcite @xcite . planner-augmented llms. given llms' poor performance in classical planning tasks @xcite @xcite , new approaches extend llms by incorporating classical planning techniques @xcite . some techniques frame the problem as a machine translation task, transforming a natural language text description of a planning problem into pddl to use with classical planners @xcite @xcite . we refer to this line of work as "planner-augmented llms." other similar approaches translate natural language into alternative representations of the planning problem, such as finite state automata @xcite or logic programming @xcite , to solve them. benchmarking llms on planning tasks. significant efforts have been made to develop benchmarks to assess the use of "llms as planners." planbench @xcite evaluates llms on various planning-related tasks, including plan generation, cost-optimal planning, plan verification, and others. their work focuses on evaluating llms on their ability to generate plans. in contrast, planetarium focuses on evaluating "planner-augmented llms". here, the goal is to determine whether an llm can successfully translate natural language descriptions of planning problems into the correct pddl representations of those problems. this is arguably a more natural use of llms, but it is harder to evaluate because the output is an internal representation and not the end result of the planning process. ## preliminaries to present planetarium, we first introduce planning, pddl, and scene graphs. ## evaluation framework in this section, we describe the design of our benchmark. planetarium consists of two components: an algorithm that validates whether two pddl problem files, a ground truth file and an llm-generated file, are equivalent, and a curated dataset of planning tasks against which an llm can be evaluated. ## evaluating llms on planetarium as an initial snapshot of the field's current state, we evaluate several api-access and open-weight language models on planetarium in both zeroshot and fine-tuned settings. we find that while powerful models like gpt-4o can often generate valid pddl problems (in the sense that some plan solves them), they are rarely correct. this result underscores the need for planetarium's rigorous approach to evaluation of pddl generation. the code to recreate the entire evaluation is available at github.com/batsresearch/planetarium. ## conclusion planetarium is a new benchmark for assessing the ability of llms to translate natural language descriptions of planning problems into pddl. our work reveals that language models find it challenging to generate semantically correct structured planning language descriptions. models like gpt-4o can often produce valid, seemingly correct descriptions (94.4%) when, in reality, only a small fraction (24.8%) are genuinely correct. we hope that planetarium will drive progress on hybrid approaches combining llms and classic planners, setting a standard for evaluating such tasks.