T ext2 W orld: Benchmarking Large Language Models for Symbolic World Model Generation (2025)

Abstract: Recently, there has been growing interest in leveraging large language models (LLMs) to generate symbolic world models from textual descriptions. Although LLMs have been extensively explored in the context of world modeling, prior studies encountered several challenges, including evaluation randomness, dependence on indirect metrics, and a limited domain scope. To address these limitations, we introduce a novel benchmark, Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation. We benchmark current LLMs using Text2World and find that reasoning models trained with large-scale reinforcement learning outperform others. However, even the best-performing model still demonstrates limited capabilities in world modeling. Building on these insights, we examine several promising strategies to enhance the world modeling capabilities of LLMs, including test-time scaling, agent training, and more. We hope that Text2World can serve as a crucial resource, laying the groundwork for future research in leveraging LLMs as world models.

## introduction the significance of world models for intelligent behavior has been historically acknowledged in early psychological theories, which posited that organisms employ internal representations of the external world for prediction and planning @xcite . furthermore, @xcite extends this concept by highlighting world modeling as a core component of autonomous machine intelligence. in this paper, we primarily study symbolic world models (also known as domain models), which are formal representations of an environment's dynamics and * equal contribution. † corresponding author constraints. in recent years, large language models (llms) @xcite @xcite have showcased their understanding of common-world knowledge, making them promising candidates for generating symbolic world models, which requires inferring action dynamics and constraints from solely natural language description @xcite @xcite . some works have already explored this across numerous tasks, including planning @xcite , game design @xcite , reinforcement learning @xcite among others. despite extensive exploration, previous work for evaluating symbolic world model generation suffers from several key limitations: (i) limited domain scope: these studies are often confined to a narrow set of domains (typically fewer than 20), which limits the generalizability and applicability of their findings @xcite @xcite . (ii) evaluation randomness: some works rely on llm-based evaluation methods, which may introduce additional margins of error @xcite . preliminary experiments in section 3.6 demonstrate that the llm-based evaluation exhibits a low interannotator agreement with human annotators (cohen's κ = 0.10). (iii) indirect evaluation: some studies evaluate world models based on end-to-end success rates in model-based planning, making it difficult to identify specific failure modes @xcite . motivated by these issues, this paper introduces a novel benchmark text2world based on the planning domain definition language (pddl), as illustrated in figure 1 . specifically, to address the first issue, we initially gathered a broad set of domains, which were then filtered through an automated pipeline and manually curated to ensure their quality, ultimately resulting in a collection of hundreds of diverse domains. furthermore, to tackle the second issue, we designed multi-criteria, execution-based metrics to ensure a more robust assessment. specifically, we not only employed structural similarity for an overall evaluation but also designed component-wise f1 scores to assess finergrained aspects such as action dynamics. moreover, to overcome the third issue, we systematically applied these metrics to assess the generated world model directly, eliminating reliance on indirect feedback mechanisms. we also performed data contamination analysis using n-gram matching @xcite , revealing a lower contamination rate (µ = 0.04) compared to prior works @xcite , indicating that text2world effectively evaluates llms' world modeling capabilities rather than pattern memorization. we used text2world to benchmark the world modeling capabilities of 16 different llms from 9 model families. experimental results in to facilitate further research, benchmark and code are available at this url. ## benchmark construction the overall process of benchmark construction is shown in figure 2 . in this section, we provide a detailed explanation of each stage. ## related work neural world modeling is a long-standing research topic with widespread applications across various fields, including reinforcement learning @xcite , robotics @xcite , and autonomous driving @xcite , among others. in recent years, llms trained on massive datasets have demonstrated zero-shot capabilities across a variety of tasks, including planning @xcite @xcite , robotics @xcite , analog design @xcite , and more. preliminary studies propose directly using llms as world models @xcite @xcite , by taking the state and action as input and predicting the next state, but the unreliability and limited interpretability of llm outputs can lead to accumulating errors. moreover, some studies have shown that autoregressive models perform poorly in predicting action effects @xcite . tree-planner @xcite instead proposes to constructing the possible action space using llms before executing. another line of work focuses on leveraging llms to construct symbolic world models @xcite @xcite @xcite @xcite . for example, @xcite uses llms to generate a pddl domain model and relies on human feedback to correct errors. agentgen @xcite synthesizes diverse pddl domains, aiming to create high-quality planning data. @xcite propose to finetune llms for predicting precondition and effect of actions. despite the growing interest in this research direction, there is currently a lack of a comprehensive benchmark in this area. ## conclusion we present text2world, a novel benchmark consisting of hundreds of domains designed to evaluate the world modeling capabilities of large language models (llms). developed through a meticulous and thorough process, text2world provides a robust foundation for analysis. additionally, we conducted an extensive evaluation involving 16 different llms from 9 model families based on text2world. we hope that text2world will inspire future research in leveraging llms as world models.