C o E x – Co-evolving World-model and Exploration (2025)

Abstract: Planning in modern LLM agents relies on the utilization of LLM as an internal world model, acquired during pretraining. However, existing agent designs fail to effectively assimilate new observations into dynamic updates of the world model. This reliance on the LLM’s static internal world model is progressively prone to misalignment with the underlying true state of the world, leading to the generation of divergent and erroneous plans. We introduce a hierarchical agent architecture, CoEx, in which hierarchical state abstraction allows LLM planning to co-evolve with a dynamically updated model of the world. CoEx plans and interacts with the world by using LLM reasoning to orchestrate dynamic plans consisting of subgoals, and its learning mechanism continuously incorporates these subgoal experiences into a persistent world model in the form of a neurosymbolic belief state, comprising textual inferences and code-based symbolic memory. We evaluate our agent across a diverse set of agent scenarios involving rich environments and complex tasks including ALFWorld, PDDL, and Jericho. Our experiments show that CoEx outperforms existing agent paradigms in planning and exploration.

## introduction while recent advances in large language model (llm) capabilities have enabled significant performance gains in llm-based agents, such agents fall short when tasked with achieving goals in novel environments with limited prior information. meanwhile, humans particularly excel in such scenarios, actively exploring environments and adapting plans based on new observations @xcite . with exploration, the understanding of the environment, or world model @xcite , is continuously adapted, and planning is grounded on such understanding. we attribute the limitations of existing llm agents like react @xcite and reflexion @xcite in novel environments, to their monolithic design, which entangles planning, reasoning, and action generation within a single llm instance. instantiated through in-context learning (icl) with action-level few-shot exemplars, this design faces two fundamental limitations: with this bias, action-level planning results in a myopic horizon, limiting generalization to longer horizon tasks. in contrast, we argue that agents need both the ability to plan exploratory goals, and the ability to directly leverage the outcome of exploration to update an adaptive world model. we introduce our proposed framework, coevolving world-model and exploration (coex), which formulates llm agent planning as a state machine over a subgoal-level belief state. our first distinction is a planner which conditions on a subgoal-level belief state, rather than low-level actions. this enables coex to leverage subgoals to plan deliberate exploration goals when faced with uncertainty. an additional flexibility of our approach is dynamic planning based on exploration, where subgoals can be generated anew when the agent deems the current plan suboptimal. our second distinction is a robust mechanism for world model co-evolution through a process of verification and synthesis. following each subgoal attempt, coex distills task-relevant insights from raw experience, using llm-based filtering to generate targeted updates to an adaptive belief state. specifically, we propose a neurosymbolic belief state design, combining: through this unified subgoal-level planning and exploration strategy, coex achieves co-evolving agent planning and adaptive world modeling, addressing the core limitations of monolithic llm designs. to demonstrate the effectiveness of coex, we evaluate our proposed method on a diverse set of challenging agent planning scenarios including alfworld @xcite , jericho @xcite , and pddl @xcite domains. ## related work in fig. 1 , we illustrate the landscape of existing llm agent paradigms along the dimensions of planning granularity and world model adaptation. as overviewed in section 1, existing llm agents with static world model and myoptic action-level planning fall in the lower-left (fig. 1 ), react @xcite and reflexion @xcite , relying on a monolithic agent design which conflates planning, reasoning, and action generation into a singular llm agent. expel @xcite implements an offline form of world model updates, but preclude real-time adaptation. wall-e @xcite proposes offline rule learning for world model alignment, extracting symbolic rules from collected trajectories, but similarly does not support online adaptation. toward the desired upper-right, where ours is placed, existing efforts can be categorized by two directions. first, world model update (upper left): the upper-left shows approaches leveraging world model for planning, repurposing llms as world models directly @xcite , or using llms to generate code-based world models @xcite . in particular, the latter can represent adaptive knowledge as code @xcite in an online manner. however, by llm strictly taking the role of code generator, llm reasoning abilities cannot be leveraged for planning, instead delegating planning to external algorithms such as mcts. in addition, they focus entirely on modeling action-level dynamics of the world, and in practice are limited to demonstrating basic competence in complex planning tasks such as alfworld @xcite . second, subgoal-level planning (lower right): adaplanner @xcite introduces dynamic plan improvement based on episodic and execution feedback, but rely on handcrafted examples to demonstrate plan adaptation in detail, and do not support learning with an explicit world model. hiagent @xcite employs subgoal-level planning, but similarly relies on implicit world models within monolithic agents, using subgoals as memory chunks and observation summarization to reduce context redundancy. ## background in this section, we discuss the notions of exploitation bias and limited adaptation within the context of llm agents. we begin with a general reinforcement learning (rl) formulation of llm agent's in- ## conclusion we study the problem of overcoming the limitations of existing llm agents, namely, exploitation bias and limited adaptation. we propose coex, enabling subgoal-driven exploration, allowing its internal world model to co-evolve with real-time observations. this is facilitated through a neurosymbolic belief state that integrates textual reasoning with symbolic memory, ensuring that the agent's understanding remains aligned with the true state of its environment. our evaluations on benchmarks requiring planning and exploration, including alf-world, jericho, and pddl, show that coex outperforms strong baselines across diverse tasks.