Situated Dialogue Learning through Procedural Environment Generation (2022)

Abstract: We teach goal-driven agents to interactively act and speak in situated environments by training on generated curriculums. Our agents operate in LIGHT (Urbanek et al. 2019)—a large-scale crowd-sourced fantasy text adventure game wherein an agent perceives and interacts with the world through textual natural language. Goals in this environment take the form of character-based quests, consisting of personas and motivations. We augment LIGHT by learning to procedurally generate additional novel textual worlds and quests to create a curriculum of steadily increasing difficulty for training agents to achieve such goals. In particular, we measure curriculum difficulty in terms of the rarity of the quest in the original training distribution—an easier environment is one that is more likely to have been found in the unaugmented dataset. An ablation study shows that this method of learning from the tail of a distribution results in significantly higher generalization abilities as measured by zero-shot performance on never-before-seen quests.

## introduction a key hypothesis in the pursuit towards creating goal-driven natural language-based agents posits that interactivity and environment grounding is critical for effective language learning @xcite @xcite . text games provide a platform on which to interactively train agents that can both act and speak in a situated manner-producing language that is both goal-driven and contextually relevant. agents in text games operate-perceiving, acting in, and speaking to others in a world-entirely using textual natural language. these games are structured generally as sequential decision making problems in the form of puzzles or quests that must be completed to advance in the game. as seen in figure 1 , we focus on creating agents in light @xcite , a large-scale crowdsourced fantasy text-adventure game, consisting of rich textual worlds-locations, objects, and characters with personas, and quests-motivations for each character. to complete these quests, an agent must: (1) maintain character via its persona; and (2) reason in a partially observable world about potential actions and utterances based on incomplete descriptions of the locations, objects, and other characters. this requires several human like competencies such as commonsense reasoning, dynamic natural language understanding, and operating in combinatorially sized language-based stateaction spaces. although recent work has provided evidence showing that interactive language learning via reinforcement learning (rl) in text games can be significantly more sample efficient than static supervised learning @xcite when creating goal-driven natural language agents, their ability to robustly generalize to novel scenarios is limited. in sequential decision making problems in particular, this generalization gap is the result of an agent simply memorizing trajectories, e.g. the sequence of actions and dialogues required to finish a game, and thus being unable to react in novel scenarios-i.e. the agent learns from the head the training data and simply memorizes the long tail. one way of decreasing this generalization gap is by training agents on procedurally generated environments-wherein the agent learns a family of parametrized tasks with a significantly larger state-action spaces than singular environments, thus effectively making the memorization of trajectories impossible @xcite . drawing inspiration from all of these ideas, we create a method that learns to create a training curriculum of increasingly more difficult novel procedurally generated environments. our contributions are threefold: (1) we present a method of parametrizing and generating a curriculum of environments in text games; (2) we show how to effectively train reinforcement learning agents on this curriculum; and (3) provide an experimental study showing that our method enables significantly better generalization than those training on singular environments. ## procedural environment generation this section describes our procedural generation pipeline as seen in figure 2 , starting with world and quest generation, followed by aligning both of them. there are two main kinds of models that we use for the different modules in this pipeline: retrieval and generative. the light questing environment. the light game environment @xcite foot_0 is a multi-user fantasy text-adventure game consisting of a rich, diverse set of 1775 characters, 663 locations, and 3462 objects. characters are able to perform templated actions to interact with both objects and characters, and can speak to other characters through free form text dialogues. actions in text games generally consist of verb phrases (vp) followed optionally by prepositional phrases (vp pp). for example, get obj, put obj, give obj to char, etc.. these actions change the state of the world which is expressed through text descriptions. quests in light @xcite take the form of a short motivation and goal action that is required reach the world state required to finish the game. for example, if the short motivation is "your motivation is to acquire a sword", then the corresponding goal state would be for the character to have a sword in their inventory and goal action would be get sword. this environment also contains a set of human expert demonstration of people speaking and acting in character while playing the quests mentioned above. further details are found in appendix a.1. ## curriculum learning generating curriculums. we generate curriculums by building off of our procedural light game instance generation pipeline. we make the observation that the original quests in light are heavily skewed towards certain quest types-with the majority involving goals and short motivations that contain objectives related to getting an object, and hitting or hugging another character (figure 3 ). we further note that the first verb in the short motivation forms the basis of the quest for that agent. actions in light, and more generally in text games, are executed in the game engines on the basis of verbs-engine subroutines are linked to verbs with nouns forming arguments-and as such are primarily responsible for changing the state of the world. for example, get sword invokes the get subroutine that places an object, in this case a sword, in the character's surrounding into their inventory. as the quest is generated early in the pipeline, with the world and the rest of the components being conditioned on it, we can say that the first verb in the short motivation is an important dimension along which we can assess the distribution of individual light game instances. thus, concretely, the verb counts from the short motivation aggregated over a set of quests represents the primary dimension along which we measure the distribution of quests. parametrizing curriculum difficulty. given the relative imbalance of this multinomial distribution, as seen in figure 3 , we hypothesize that a light agent only learns to do well on certain types of objectives and not others-memorizing trajectories for less seen quest types, i.e. those found in the tail of the distribution. preliminary evidence for this hypothesis is also seen in prabhumoye et al. (2020), where they show a positive correlation between the number of instances of a particular type of quest during training and the final test goal-achievement performance. based on these observations and our initial hypothesis, we use this particular dimension to parametrize curriculum difficulty for training light agents-quest types that are rarer in the initial training data will be harder for the agent to generalize to in a zero-shot setting. intuitively, we seek to create curriculums that contain a diverse set of game instances with quest types that are not often found in the initial training data. our earlier observations let us hypothesize that this will enable the light agent to more effectively learn from rare instances of quests as op- figure 6 shows that decreasing n has the intended effect of decreasing imbalance with respect to verb types. generating using this pipeline has the added effect of increasing diversity within the pool of each available quest type. one measure of diversity within the pool of a single quest type is the types of nouns contained within the short motivations-these generally correspond to the characters, locations, and objects mentioned. figure 6 shows that decreasing imbalance in the verb types for a short motivation also results in decreasing imbalance in noun types, once again corresponding to decreasing n. short motivation generation is one of the first steps in the pipeline, i.e. the rest of the pipeline is conditioned on it, and as such increasing the flatness of the distribution there has the effects of increasing distribution for downstream components. ## evaluation we conduct two separate evaluations: the first measures the effectiveness of the various models in the procedural environment generation pipeline as well as the effectiveness of the pipeline as a whole. the second provides zero-shot ablations of the light rl agents trained on the resulting curriculums and answers the questions (1) how does the relative difficulty of the training quests effect test performance?; (2) how does the diversity of the environments during training effect test performance?; and ## related work text-based game playing and generation. recent text game playing works have focused on tackling three primary challenges: (1) how to represent agent knowledge to effectively operate in partially observable environments @xcite ; (2) scaling rl algorithms to handle combinatorial natural language state-action spaces @xcite @xcite ; and (3) giving agents commonsense priors to better reason about the world @xcite on the flip side, we have procedural generation of games with works such as @xcite ; @xcite ; khalifa et al. (2020) that focus on creating content especially for 2d visual games via search or reinforcement learning based methods. @xcite use knowledge graphs to ground language and produce worlds and quests separately for text games from existing corpora such as stories. @xcite leverage light to learn to generate interactive fiction worlds on the basis of locations, characters, and objects-this work is closest in spirit to our own world generation module later on. they all focus on either generating or playing games. goal oriented dialogue. sub-tasks within the overall task of goal oriented dialogue, such as dialogue state management @xcite @xcite and response generation @xcite have used rl to boost performance. as noted by @xcite , the negotiation tasks of @xcite , where two agents are trying to convince each other to perform certain actions, are related to the tasks in light-quests. these works all lack environment grounding. curriculum learning. curriculums in reinforcement learning have traditionally been used to set goals of steadily increasing difficulty for an agent @xcite . the difficulty of these curriculums are generally measured difficulty via proxy of agent performance @xcite )-methods either choose to adversarially set goals of steadily increasing difficulty @xcite @xcite or to maximize learning performance based on environment instances an agent finds difficult historically @xcite . while we were inspired by these works, they all focus on searching for goals for agents which can be difficult to scale to complex tasks such our own natural language motivation-based goals. we'd also like to note that most works using procedural generation to benchmark rl agents such as @xcite ; küttler et al. (2020); samvelyan et al. (2021) rely on the underlying richness of the game engines to generate novel environments as opposed to learning to generate. ## conclusions we focus on the problem of improving zero-shot generalization abilities of goal-driven rl agents to act and speak via natural language. an (obviously) key component of achieving this is to train the rl agents on a balanced training dataset that matches the test data in distribution. as this is an unlikely scenario in most real-world applications, we make the observation that we can artificially augment our pool of training environments by generating curriculums to mimic this. in our text game domain, with goal-driven situated natural language agents, we hypothesize-and gather supporting evidence suggesting-that an effective way to parametrize such distributions is by looking at the primary verbs within an agent's motivation and bringing the distribution of verb types as close to uniform as possible. curriculum training significantly increases an agent's ability to generalize to novel scenarios. as noted by @xcite and @xcite , the ability to speak and act in these textual fantasy worlds has implications for domains beyond text-games. text games are a platform where agents can interact in a relatively isolated environment and learn to interactively communicate effectively through natural language in a situated manner. our methods use both large language models and deep reinforcement learning and are prone to the pitfalls that other contemporary methods using these techniques face, especially in the areas of dialogue and text game systems. we mitigate this first pitfall by restricting our current system to a retrieval based dialogue, ensuring that we can filter out non-normative dialogue usages beforehand, though we will note that the system can be extended to generative systems as described in @xcite . further, the light dataset is crowdsourced and contains data biases that can be attributed to the crowdworkers tasked with creating the data. @xcite provides an in depth discussion regarding the inherent dataset biases, such as gender bias in the distribution of characters, in light and techniques to mitigate them-we follow these methods to reduce their effects on both the environment generation and agent training procedures.