Enhancing Agent Learning through World Dynamics Modeling (2024)

Abstract: Large language models (LLMs), trained on vast amounts of internet data, have developed a broad understanding of the world, enhancing the decision-making capabilities of embodied agents. This success is largely due to the comprehensive and in-depth domain knowledge within their training datasets. However, the extent of this knowledge can vary across different domains, and existing methods often assume that LLMs have a complete understanding of their environment, overlooking potential gaps in their grasp of actual world dynamics. To address this gap, we introduce Di scover, V erify, and E volve (DiVE), a framework that discovers world dynamics from a small number of demonstrations, verifies the correctness of these dynamics, and evolves new, advanced dynamics tailored to the current situation. Through extensive evaluations, we analyze the impact of each component on performance and compare the automatically generated dynamics from with human-annotated world dynamics. Our results demonstrate that LLMs guided by can make better decisions, achieving rewards comparable to human players in the Crafter environment.

## introduction by absorbing internet-scale knowledge autoregressively, large language models (llms) develop a broad understanding of the world @xcite @xcite . this understanding enables them to perform well across a variety of tasks @xcite @xcite . however, achieving this level of comprehension requires a training dataset that is diverse, * equal contribution. since llms tend to follow the most common patterns in the dataset @xcite and are prone to hallucination, it is crucial that the necessary information not only appears frequently on the internet but is also reliable @xcite . for tasks and environments that are newly emerging @xcite , these requirements are often difficult to meet due to the noisy nature of internet data, leading to potential knowledge gaps. while llms may have a general understanding of a domain, optimal decision-making requires indepth, state-specific knowledge. providing llms with such tailored information enhances their grasp of both the environment and the current state. for example, mastering go involves not just knowing the rules but also applying strategies for specific board states. this specialized knowledge varies across domains and states, making offline collection impractical. to address these challenges, we propose discover, verify, and evolve (dive). building on the concept of the world model @xcite , dive not only discovers and verifies primitive world dynamics from demonstrations but also evolves state-specific knowledge for the downstream domain. by providing llms with this comprehensive set of dynamics, dive bridges the knowledge gap between llms and the downstream domain, thereby enhancing their decision-making abilities. dive consists of three distinct components: the discoverer: this component iteratively uncovers the environment's dynamics from demonstrations using a curriculum learning approach. the verifier: this component eliminates unreliable dynamics caused by llms' tendency to hallucinate. the evolver: this component reasons through in-depth, state-specific strategies tailored to the current situation based on the learned dynamics. in the crafter environment @xcite and the minihack environment @xcite , dive learns comprehensive and reliable dynamics from demonstrations, guiding the agent's decision-making process by evolving in-depth strategies. this enables the agent to outperform all baselines, achieving rewards comparable to human players in the crafter environment and surpassing the performance of methods requiring task-specific training in the minihack environment. to gain deeper insights into dive's behavior, we provide both quantitative and qualitative analyses. in summary, our primary contribution is a framework that learns world dynamics from demonstrations, guiding llms in the decision-making process by online evolving contextual strategies. this approach bridges potential knowledge gaps, resulting in a more optimal decision-making process for llms. ## the knowledge gap in this work, we consider a setting where a pretrained llm-based agent is employed to solve tasks in downstream domains. conceptually, we define k llm as the set of knowledge embedded in the llms through their training process, and k target as the universal set of knowledge relevant to the downstream domains. to ensure the effectiveness of the llms, we aim for k relevant , the subset of k llm that is relevant to k target , to cover the broadest possible scope. additionally, we seek for k relevant to contain more reliable knowledge k + than inaccurate knowledge k -, where k @xmath0. thus, we define three desirable properties to guide our system design: we recognize that precisely quantifying the knowledge overlap between llms and a downstream domain is challenging. in section 3, we provide a mathematical analysis showing how dive improves across all three dimensions. in section 5.4, we measure the knowledge overlap to demonstrate that dive effectively learns useful game dynamics and, to some extent, bridges the knowledge gap. ## dive: discover, verify, and evolve in an ideal scenario, one could bridge the knowledge gap by fine-tuning the llm to adapt to the target domain. however, this approach is often less practical due to its reliance on large amounts of annotated data and significant computational overhead @xcite @xcite . our framework, dive, is designed to address the knowledge gap while considering all three desirable properties-recall, precision, and depth-without requiring extensive data collection from the target domain. it is a prompt-based method that learns world dynamics w directly from the environment. as shown in figure 2 , dive is initially bootstrapped using a small set of human demonstration trajectories h, each consisting of observations o t , actions a t , and rewards r t at each timestep t. we then transform each observation o t into it is daytime\nstate description: \n-east: immediate (grass); nearby (unexplored_area); \n-north: immediate (grass); nearby (unexplored_area); \n-north east: immediate (grass); nearby (cow, unexplored_area); \n-north west: immediate (grass); nearby (unexplored_area, tree); \n-south: immediate (grass); nearby (unexplored_area); cow: north east 5 blocks away (nearby) (objects in between: {'grass'}) \n-facing grass on the south.\nyour status:\n-health: 9/9\n-food: 9/9\n-drink: 9/9\n-energy: 9/9\nyou have nothing in your inventory. it is daytime\nstate description: \n-east: immediate (grass); nearby (unexplored_area); \n-north: immediate (grass); nearby (unexplored_area); \n-north east: immediate (grass); nearby (cow, unexplored_area); \n-north west: immediate (grass); nearby (unexplored_area, tree); \n-south: immediate (grass); nearby (unexplored_area); cow: north east 5 blocks away (nearby) (objects in between: {'grass'}) \n-facing grass on the south.\nyour status:\n-health: 9/9\n-food: 9/9\n-drink: 9/9\n-energy: 9/9\nyou have nothing in your inventory. it is nighttime\nstate description: \n-east: immediate (sand); nearby (path, grass); distant (tree, unexplored_area); \n-north: immediate (zombie); nearby (path, sand, grass); distant (tree, unexplored_area); \n…… -zombie: north 1 blocks away (immediate) (objects in between: none) \n-table: south east 9 blocks away (nearby) (objects in between: {'path', 'sand', 'water', 'grass'}) \n-facing zombie on the north.\nyour status:\n-health: 2/9\n-food: 8/9\n-drink: 8/9\n-energy: 3/9\nyour inventory:\n-wood: 1\nstone: 9\n-coal: 4\n-wood_pickaxe: 1\n-stone_pickaxe: 1\n-wood_sword: 1\n-stone_sword: 1\n it is nighttime\nstate description: \n-east: immediate (sand); nearby (path, grass); distant (tree, unexplored_area); \n-north: immediate (zombie); nearby (path, sand, grass); distant (tree, unexplored_area); \n …… -zombie: north 1 blocks away (immediate) (objects in between: none) \n-table: south east 9 blocks away (nearby) (objects in between: {'path', 'sand', 'water', 'grass'}) \n-facing zombie on the north.\nyour status:\n-health: 2/9\n-food: 8/9\n-drink: 8/9\n-energy: 3/9\nyour inventory:\n-wood: 1\nstone: 9\n-coal: 4\n-wood_pickaxe: 1\n-stone_pickaxe: 1\n-wood_sword: 1\n-stone_sword: 1\n verbalize discover verify stone can be found near iron, coal, but it is not associated with cow, zombie, skeleton. you cannot walk directly through stone. the language space as õt using a verbalizer, resulting in trajectories represented by õt , a t , and r t at each timestep t. subsequently, the discoverer extracts a set of world dynamic candidates, @xmath1, w-}, from human demonstrations h, where w+ and wrepresent the correct and inaccurate world dynamic sets, respectively. empirically, we find that the inclusion of win w is often unavoidable, either due to llm's difficulties in extracting meaningful knowledge from trajectory data or its tendency to hallucinate. to address this, we employ the verifier to filter out potentially invalid and conflicting world dynamic candidates from w, leaving only the valid dynamics w. lastly, we use the evolver, designed to derive advanced dynamics i tailored to the verbalized observation õt based on the filtered world dynamics w. the final decision-making process on primitive actions a @xmath2, sub-tasks st , and actions a. the planning procedure is further guided by both w and i. in cases where w ̸ = ∅, recall, precision and depth are guaranteed to increase as formulated below: the dive framework can be divided into two stages: an offline dynamics learning stage and an online strategy learning stage. ## experiment setup to demonstrate the effectiveness of dive in bridging the knowledge gap, we evaluate its performance within the crafter and minihack environments. for a fair comparison, all llm-based agents, including ours, utilize the gpt-4o model and the same environment seed in the crafter setting. further details on the setups can be found in appendix a.1 and b.1. ## experimental results we evaluate the performance of dive in the crafter and minihack environments. in section 5.1, we present the overall results to demonstrate dive's effectiveness in bridging the knowledge gap in these tasks. section 5.2 provides a detailed analysis of the contributions of individual components through controlled experiments. then, in section 5.3, we evaluate the effectiveness of the dynamics learned by dive in the crafter environment. finally, in section 5.4, we further analyze the learned dynamics in both the crafter environment and a modified minihack setting with altered dynamics. ## conclusion we introduce dive, a framework that bridges the knowledge gap between llms and downstream domains. the discoverer extracts world dynamics, while the verifier filters unreliable candidates. in an online setting, the evolver reasons strategies through interaction. our experiments demonstrate that dive effectively bridges the knowledge gap between llms and downstream domains.