Language-guided World Models: A Model-based Approach to AI Control (2024)

Abstract: Developing internal world models for artificial agents opens an efficient channel for humans to communicate with and control them. In addition to updating policies, humans can modify the world models of these agents in order to influence their decisions.The challenge, however, is that currently existing world models are difficult for humans to adapt because they lack a natural communication interface. Aimed at addressing this shortcoming, we develop *Language-Guided World Models* (LWMs), which can capture environment dynamics by reading language descriptions. These models enhance agent communication efficiency, allowing humans to simultaneously alter their behavior on multiple tasks with concise language feedback. They also enable agents to self-learn from texts originally written to instruct humans. To facilitate the development of LWMs, we design a challenging benchmark based on the game of MESSENGER (Hanjie et al., 2021), requiring compositional generalization to new language descriptions and environment dynamics. Our experiments reveal that the current state-of-the-art Transformer architecture performs poorly on this benchmark, motivating us to design a more robust architecture. To showcase the practicality of our proposed LWMs, we simulate a scenario where these models augment the interpretability and safety of an agent by enabling it to generate and discuss plans with a human before execution. By effectively incorporating language feedback on the plan, the models boost the agent performance in the real environment by up to three times without collecting any interactive experiences in this environment.

## introduction model-based agents are artificial agents equipped with probabilistic "world models" that are capable of foreseeing the future state of an environment @xcite . world models endow these agents with the ability to plan and learn in imagination (i.e., internal simulation) and have led to exciting results in the field of reinforcement learning @xcite @xcite . these models have been studied extensively for the purpose of improving the autonomous performance of artificial agents. in this paper, we endorse and enhance the model-based approach for a different goal: to strengthen the controllability of artificial agents. since all policies of a model-based agent are optimized with respect to a common world model, a human can adjust multiple policies simultaneously by making appropriate changes to this model. this mechanism complements the model-free approach that updates policies individually, offering greater efficiency and flexibility in control. for example, by incorporating the fact that the floor is slippery into the world model of a robot, a person can effectively remind it to handle every object in a room with greater caution. if the performance of the robot on a task remains unsatisfactory, the person can continue to fine-tune its policy for that specific task. in contrast, without a world model, they have to separately adapt the robot's policies to the slippery-floor condition. the model-based approach requires world models that can be easily modulated by humans. traditional world models fall short in this quality because they can only be modified using observational data, which is not a suitable medium for humans to convey intentions @xcite . to overcome the limitations of these models, we develop language-guided world models (lwms)-world models that can be effectively steered through human verbal communication. agents equipped with lwms inherit all the benefits of model-based agents while being able to incorporate language-based supervision. this capability reduces human teaching effort and mitigates the risk of agents taking harmful actions in an environment to explore its dynamics. lwm-based agents can also self-improve by reading "free" texts (a) we illustrate a potential application of lwms to improving ai safety and transparency. these models enable an agent to generate visual plans and invite a human supervisor to validate them. moreover, the human can adjust the plans by modifying the agent's world model with language feedback, in addition to directly correcting its policy. (b) we design an architecture for lwms that exhibits strong compositional generalization. we replace the cross-attention mechanism of the standard transformer with a new attention mechanism inspired by @xcite to effectively incorporate language descriptions. we then train a model that auto-regressively generates tokenized observations conditioned on language descriptions and actions. building lwms poses a unique research challenge: grounding language to environmental dynamics. this problem is difficult because the language used to describe environment dynamics can be incredibly rich and complex, encompassing a wide range of concepts such as entity names, appearances, motions, interactions, spatial and temporal relations, and more. moreover, in natural settings, especially when describing artificial environments (e.g., games), new concepts are often introduced but may not always be clearly defined. humans deal effectively with this issue because they possess remarkable reasoning capabilities that allow them to infer word meanings from observations. for example, a caption like "the ziff, which is chasing the player, is extremely hostile" and a video depicting this scene likely provide enough clues for a person to determine what "the ziff" refers to, assuming that they are familiar with the concept of "chasing". not only understanding word meanings, humans are also capable of applying newly learned words in novel ways, enabling imagination of new dynamics, such as envisioning a "fleeing ziff" that runs away from the player. toward building world models with similar capabilities, we construct a benchmark based on the game of messenger @xcite . in this benchmark, a model is given trajectory "videos" of games involving several entities interacting with each other. each video is accompanied by language descriptions of the attributes of the entities. the model begins with almost zero language understanding and has to identify the entities and learn the grounded meanings of their attributes purely by watching the videos. at test time, it must demonstrate compositional generalization by being able to simulate environments featuring entities with attributes different from those it observes during training. for example, it has to portray a "fleeing mage" despite having only seen the mage chase the player in training games. we design three evaluation settings that test for incrementally greater degree of compositional generalization. despite its apparent simplicity, our benchmark covers many complications in building robust lwms. we find that the prominent transformer model @xcite struggles in the harder evaluation settings. even with a ground-truth disentangled representation of the observations, the model cannot learn generalizable grounding functions and yields minimal improvements in simulation quality compared to a model that ignores the language descriptions entirely. we augment the model with the emma attention @xcite , which mimics a two-step reasoning process. our results confirm the effectiveness of this new architecture, as it robustly generalizes even in the hardest evaluation setting, outperforming baselines by substantial margins in various evaluation metrics. it is even competitive with a skyline model with an oracle semantic parsing and grounding capability. last but not least, we illustrate a promising application of lwms by simulating a cautious agent that, instead of performing a task right away, uses its lwm to generate an execution plan and asks a human to review it (figure 1a ). this form of preexecution communication can potentially improve the agent's safety and transparency, following the spirit of the guaranteed safe ai approach proposed by @xcite . moreover, it allows the human to improve the performance of the agent by revising the plan. in this setting, our lwm-based agent has the advantage of being able to assimilate language feedback describing the environment dynamics. we demonstrate that the language understanding capabilities of our proposed lwm are sufficient to enact this strategy. in the most challenging evaluation setting, without gathering additional interactions in the environment, the agent equipped with our model achieves an average reward three to four times higher than that of an agent using an observational world model. we hope that our work will serve as a catalyst for exploring novel approaches to developing robust language-guided world models. more generally, we call for the design of modular agents whose components are parameterized by natural language. as previously argued, a modular design can dramatically boost communication efficiency, because the same component may be involved in the learning of various policies. we hypothesize that this approach can potentially surpass the efficiency of the currently prevalent approach that integrates language into a monolithic policy (e.g, @xcite al. (2016); misra et al. (2018); anderson et al. (2018); narasimhan et al. (2018 ## background: world models we consider a markov decision process (mdp) environment e with state space s, action space a, and transition function m : s × @xmath0, 1}), where ∆ denotes the set of all probability distributions over a set. an agent implementing a policy π(a | s) : @xmath1. taking an action a t ∼ π(s t ) in state s t transitions the agent to a new state s t+1 , and incurs a reward r t+1 and a termination signal d t+1 , where s t+1 , r t+1 , d t+1 ∼ m (s t , a t ). a (one-step) world model m θ @xcite @xcite is an approximation of m (s t+1 , r t+1 , d t+1 | s t , a t ). a model-based agent uses data gathered in the environment to construct a world model and leverages it to learn policies for accomplishing tasks. foot_0 in contrast, a model-free agent learns its policies directly from data collected in the environment. model-based agents can require less effort to adapt. because all policies of a model-based agent are derived from a shared world model, any modifications made to this model would affect all of them. this feature can be exploited to reduce human effort in controlling this type of agent. specifically, suppose we concern m tasks in the environment, necessitating m policies. if there is a change in the environment dynamics, a modelbased agent only needs task-agnostic data to replicate this change in its world model. it can then re-optimize its policies with respect to the updated model. meanwhile, a model-free agent needs to collect task-specific data to re-train all of its m policies. the data collection cost of the modelfree approach scales with m, whereas that of the model-based approach is independent of m, since the policy re-optimization step uses only data generated by the world model. observational world models. the dominant approach to world modeling learns a function m θ (s t+1 , r t+1 , d t+1 | h t ) parameterized by a neural network θ and conditioned on a history h @xmath2, r 1 , d 1 , a 1 , . . . , s t , r t , d t , a t ). we refer to this class of models as observational world models because they can be adapted with only observational data, through either in-weight learning (updating the model parameters to fit a dataset of observations), or in-context learning (plugging in a history of observations). relying on observation-based adaptation leads to two drawbacks. first, controlling these models is difficult because observations are inadequate for conveying complex, abstract human intentions. second, collecting observations requires taking real actions in the environment, which can be expensive, time-consuming, and risky. ## language-guided world models (lwms) we introduce lwms, a new class of world models that can interpret language descriptions to simulate environment dynamics. these models address the drawbacks of observational world models. they allow humans to easily adapt their behavior through natural means of communication. consequently, humans can effectively assist these models, significantly reducing the amount of interactive experiences that they need to collect in environments. in addition, these models can also leverage pre-existing texts written for humans, saving human effort to fine-tune them. ## conclusion we introduce language-guided world models, which can be adapted through natural language. we outline numerous advantages of these models over traditional observational world models. our model is still lacking in performance and the gridworld environments we experiment with severely underrepresent the real world. nevertheless, we hope that this work helps envision the potential of lwms in enhancing the controllability of artificial agents and inspires future efforts to address the compositional generalization challenge. victor zhong, austin w. hanjie, sida i. wang, karthik narasimhan, and luke zettlemoyer. 2021. silg: the multi-environment symbolic interactive language grounding benchmark. in neural information processing systems (neurips). victor zhong, tim rocktäschel, and edward grefenstette. 2020. rtfm: generalising to new environment dynamics via reading. in international conference on learning representations. siyuan zhou, yilun du, jiaben chen, yandong li, dit-yan yeung, and chuang gan. 2024. robodreamer: learning compositional world models for robot imagination. arxiv preprint arxiv:2404.12377.