How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2021)

Abstract: We seek to create agents that both act and communicate with other agents in pursuit of a goal. Towards this end, we extend LIGHT (Urbanek et al. 2019)—a large-scale crowd-sourced fantasy text-game—with a dataset of quests. These contain natural language motivations paired with in-game goals and human demonstrations; completing a quest might require dialogue or actions (or both). We introduce a reinforcement learning system that (1) incorporates large-scale language modeling-based and commonsense reasoning-based pre-training to imbue the agent with relevant priors; and (2) leverages a factorized action space of action commands and dialogue, balancing between the two. We conduct zero-shot evaluations using held-out human expert demonstrations, showing that our agents are able to act consistently and talk naturally with respect to their motivations.

## introduction there has been a recent improvement in the quality of natural language processing (nlp) and generation (nlg) by machine learning (ml) @xcite ; and in parallel, improvement to goal-oriented ml driven agents in the context of games @xcite . however, agents that can communicate with humans (and other agents) through natural language in pursuit of their goals are still primitive. one possible reason for this is that many datasets and tasks used for nlp are static, not supporting interaction and language grounding @xcite @xcite @xcite . text-based games-where players see, act upon, and communicate within a dynamic world using natural 1 data can be found here https://parl.ai/ projects/light/ language-provide a platform on which to develop such goal-driven agents. light @xcite , a large-scale crowdsourced fantasy text-adventure game, consisting of a set of locations, characters, and objectsa possesses rich textual worlds, but without any notion of goals to train goal-driven agents. we present a dataset of quests for light and demonstrations of humans playing these quests (as seen in figures 2 and 3), providing natural language descriptions in varying levels of abstraction of motivations for a given character in a particular setting. to complete these quests, an agent must reason about potential actions and utterances based on incomplete descriptions of the locations, objects, and other characters. when a human is placed in a fantasy setting such as light, they already know that kings are royalty and must be treated respectfully, swords are weapons, etc.-commonsense knowledge that a learning agent must acquire to ensure successful interactions. to equip agents with relevant priors in such worlds, we domain-adapt the large-scale commonsense knowledge graph atomic @xcite to the light fantasy world-to build atomic-light. we then introduce a reinforcement learning (rl) system that incorporates large-scale language modeling and the above commonsense-based pretraining. we show that rl is superior to behavior cloning or other supervised training on our data; and that carefully combining pre-training with rl is superior to either. however, we find that although pre-training can be an effective tool in this setting, it requires more finesse than in the standard supervised setting. in particular, we find that simply pre-training a model on a large "generic" corpus @xcite of commonsense/language data or pre-training on the domain specific light corpus, and then fine-tuning via rl is less effective than training rl from scratch. furthermore, by ## related work we focus on four major areas of related work: text-based game-playing, goal-oriented dialogue, commonsense reasoning in language, and general language-informed rl. text-based game-playing. @xcite introduce textworld, a framework for procedurally generating text-based games via grammars, and @xcite @xcite build agents that operate in this environment-focusing on aspects such as efficient exploration and zeroshot generalization to new, procedurally generated environments. similarly, @xcite introduce jericho, a framework and series of baseline agents for interacting with human-made textgames such as zork @xcite . this resulted in agents developed by works such as @xcite , aiming to learn to execute contextually relevant actions. other works such as @xcite explore how to best factorize such text-game action spaces. none of these works consider agents with motivations and personas nor require any dialogue. goal-oriented dialogue. this form of dialogue has traditionally been closely related to specific tasks useful in the context of personal assistants with dialogue interfaces @xcite . rl has been studied for such tasks, usually to improve dialogue state management @xcite @xcite and to improve response quality @xcite . in particular, the negotiation tasks of @xcite , where two agents are trying to convince each other to perform certain actions, are related to the tasks in light-quests. these works all lack environment grounding and the notion of diverse agent motivations. commonsense reasoning in language. works such as @xcite focus on pre-training transformer-based language learning systems with large-scale commonsense knowledge graphs such as atomic @xcite and conceptnet @xcite for use in knowledge graph completion and story ending generation respectively. @xcite @xcite look at commonsense reasoning in interactive environments, with the former focusing on affordance extraction using word embeddings and the latter three on transferring text-game playing skills via pretraining using question-answering and large-scale knowledge graphs. language-informed reinforcement learning. @xcite provide an overview of rl informed by natural language. of these works, the ones most related to ours are those falling into the category of instruction following-where an agent's tasks are defined by high level instructions describing desired policies and goals @xcite . visual and embodied agents using natural language instructions @xcite @xcite or in language-based action spaces @xcite utilize interactivity and environment grounding but have no notion of agent motivations, nor make any attempt to explicitly model commonsense reasoning. perhaps closest in spirit to this work is @xcite , where they use artificially selected goals in light and train rl agents to achieve them. similarly to the others, this work does not contain the motivations provided by light-quests nor any modeling of commonsense reasoning. further, they limit their rl problem to 1 and 3-step trajectories that only involve speech, and no actions-compared to the human demonstrations in light-quests which contain both actions and speech sequences of average length 12.92. ## light-quests and atomic-light this section first provides a brief overview of the light game environment, followed by descriptions of the light-quests and atomic-light datasets used in this paper. background. the light game environment is a multi-user fantasy text-adventure game consisting of a rich, diverse set of characters, locations, and objects (1775 characters, 663 locations, and 3462 objects). characters are able to perform templated actions to interact with both objects and characters, and can speak to other characters through free form text. actions in text games generally consist of verb phrases (vp) followed optionally by prepositional phrases (vp pp). for example, get obj, put obj, give obj to char, etc.. there are 13 types of allowed verbs in light. these actions change the state of the world which is expressed to the player in the form of text descriptions. ## agents that act and speak this section describes the creation of the agents that learn to act and speak conditioned on their motivations in the light environment. the overall architecture and training are first outlined, followed by a detailed discussion on types of encoder pretraining. ## evaluation we conduct two ablation studies, (1) to compare the effects of the encoder pre-training tasks in rl settings vs. supervised behavior cloning, and ( 2 ) to analyze the interplay between actions and dialogue for self and partner act completions. human act goal @xmath0.6 as measured during the second phase of the light-quests data collection. ## conclusion operating on the hypothesis that interactivity is key to language learning, we introduce two datasets-a set of quests based on character motivations in fantasy worlds, light-quests, and a large-scale commonsense knowledge graph, atomic-lightand a reinforcement learning system that leverages transformer-based pre-training to facilitate development of goal-driven agents that can act and speak in situated environments. zero-shot evaluations on a set of novel human demonstration show that we have trained agents that act consistently and speak naturally with respect to their motivations. a key insight from our ablation study testing for zero-shot generalization on novel quests is that large-scale pre-training in interactive settings require careful selection of pre-training tasks-balancing between giving the agent "general" open domain priors and those more "specific" to the downstream taskwhereas static methodologies require only domain specific pre-training for effective transfer but are ultimately less effective than interactive methods.