World Models with Hints of Large Language Models for Goal Achieving (2025)

Abstract: Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 41.8%, 21.1%, and 9.9%, respectively.

## introduction reinforcement learning (rl) is effective when the agents receive rewards that propel them towards desired behaviors @xcite . however, the manual engineering of suitable reward functions presents substantial challenges, especially in complex environments @xcite . therefore, solving tasks with long horizons and sparse rewards has long been desired in rl @xcite . existing rl methods address this issue by supplementing the extrinsic rewards provided by the * equal contribution. † corresponding author. to address this issue, we draw inspiration from how humans solve long-horizon tasks efficiently. humans excel at breaking down overall goals into several sub-goals and strive to plan a reasonable route to accomplish these goals sequentially @xcite . these goals are often associated with specific actions or environmental dynamics and can ideally be expressed in concise natural language. for example, experienced minecraft players can naturally connect the action "obtaining iron" with its prerequisite actions "find an iron ore" and "breaking iron ore". consequently, we propose dreaming with large language model (dllm), a multi-modal modelbased rl approach that integrates language hints (i.e., goals) from llms into the rollouts to encourage goal discovery and reaching in challenging and sparse-reward tasks, as illustrated in figure 1 . dllm's world model processes visual inputs and sentence embeddings of natural language descriptions for transitions and learns to predict both. it then rewards the agent when the predicted embeddings are close enough to the goal, facilitating the agents' use of inductive bias to achieve task goals. thanks to the power of prompt-based llms, dllm can influence agents' behaviors in distinct manners based on the prompts provided for identical tasks, resulting in multiple styles of guidance for the agents. for example, when an agent needs to obtain iron in minecraft, it can be guided directly to break iron ore, explore more for a better policy, or try interpolating both strategies. empirically, we evaluate dllm on various sparse-reward environments, including homegrid @xcite , crafter @xcite , and minecraft @xcite . experimental results demonstrate that dllm outperforms recent strong methods in task-oriented and exploration-oriented environments, showcasing robust performance in guiding exploration and training of the agent within highly complex scenarios. in homegrid, crafter and minecraft environments, we successfully improve the performance by 41.8%, 21.1% and 9.9%, respectively, compared to the strongest baseline dynalang @xcite , achievement distillation @xcite , and dream-erv3 @xcite . we also observe that leveraging more powerful language models and providing the agent with comprehensive language information results in even better performance. our contributions are as follows: (a) we propose dllm, a multi-modal model-based reinforcement learning approach that utilizes human natural language to describe environmental dynamics, and incorporates llm's guidance in model rollouts to improve the agent's exploration and goal-completion capabilities; (b) based on goals extracted by llms, dllm can generate meaningful intrinsic rewards through an automatic descending mechanism to guide policy learning; (c) experimental results demonstrate that dllm outperforms recent strong baselines across diverse environments. intrinsically motivated rl. intrinsically motivated rl is proposed to encourage exploration in sparse-reward, large state-action space environments by providing additional dense rewards @xcite . @xcite use curiosity as an intrinsic reward, measuring prediction proficiency within a selfsupervised model, while @xcite propose random network distillation (rnd), which uses prediction error from a fixed random neural network and performs well in montezuma's revenge. subsequent studies improve rnd with methods like distributional modeling @xcite . there are also other approaches focused on maximizing state diversity @xcite and skill diversity @xcite . despite their success, intrinsically motivated rl methods may struggle with large state-action spaces and complex tasks since they focus on exploring novel states, not necessarily useful ones. this purposeless exploration can hinder performance. therefore, integrating meaningful guidance, such as commonsense knowledge and explicit subgoals, is crucial @xcite . dllm considers these factors with a specific focus on long-term decision-making. during rollouts, dllm applies intrinsic rewards when the agent achieves goals set by llm in previous steps, strengthening the understanding of the agent's contextual connections. leveraging large language models (llms) for language goals. pre-trained llms showcase remarkable capabilities, particularly in understanding common human knowledge. naturally, llms can generate meaningful and human-recognizable intrinsic rewards for intelligent agents. @xcite leverage pre-trained llms as task-specific priors for managing text-based metadata within the context of supervised learning. @xcite utilize llms as commonsense priors for zero-shot planning. similar efforts are made by @xcite ; @xcite ; @xcite ; @xcite , who propose diverse prompt methods and algorithmic structures to mitigate the prob-lems of hallucination and inaccuracy when employing llms directly for decision-making. @xcite examine an approach where an agent utilizes an llm as a policy that undergoes progressive updates as the agent engages with the environment, employing online reinforcement learning to enhance its performance in achieving objectives. @xcite propose to leverage the llms to guide skill chaining. @xcite , which leverages llms to generate intrinsic rewards for guiding agents, integrating llm with rl. however, the guidance obtained using this approach is only effective in the short term. dllm draws inspiration from ellm and endeavors to extend the guidance from llms into long-term decision-making. ## preliminaries we consider a partially observable markov decision process (pomdp) defined by a tuple (s, a, o, ω, p, γ, r), where @xmath0, @xmath1, and observation @xmath2, a). p (s ′ |s, a) represents the dynamics of the environment, r and γ are the reward function and discount factor, respectively. during training, the agent's goal is to learn a policy π that maximizes discounted cumulative rewards, i.e., max e π ∞ @xmath3, a t ) . additionally, we define two sets of natural language sentence embeddings: the set of sentence embeddings for transitions, denoted as u , and the set of sentence embeddings for goals, denoted as g. in this context, each @xmath4, while @xmath5. we permit the llm to output any content within specified formats, thereby enlarging the support of the goal distribution to encompass the space of natural language strings. thus, g should encompass all possible @xmath6, i.e., u ⊂ g. we also define the goal-conditioned intrinsic reward function r int (u|g), and the dllm agent optimizes for an intrinsic reward r int alongside the reward r from the environment. assuming that goals provided by natural language are diverse, commonsense sensitive, and context-sensitive, we expect that maximizing r int alongside r ensures that the agent maximizes the general reward function r without getting stuck in local optima. ## dreaming with llms this section systematically introduces how dllm obtains guiding information (goals) from llms and utilizes them to incentivize the agent to manage long-term decision-making. ## experiments the primary goal of our experiments is to substantiate the following claim: dllm helps the agent by leveraging the guidance from the llm during the dreaming process, thereby achieving improved performance in tasks. specifically, our experiments test the following hypotheses: baselines. since we include natural language information in our experiments, we consider employing ellm @xcite and dynalang @xcite as baselines. 1 we also compare against other recent strong baseline algorithms that do not utilize natural language in each environment. ## conclusion and discussion we propose dllm, a multi-modal model-based rl method that leverages the guidance from llms to provide hints (goals) and generate intrinsic rewards in model rollouts. dllm outperforms recent strong baselines in multiple challenging tasks with sparse rewards. our experiments demonstrate that dllm effectively utilizes language information from the environment and llms, and enhances its performance by improving language information quality.