Learning to Generalize for Sequential Decision Making (2020)

Abstract: We consider problems of making sequences of decisions to accomplish tasks, interacting via the medium of language. These problems are often tackled with reinforcement learning approaches. We find that these models do not generalize well when applied to novel task domains. However, the large amount of computation necessary to adequately train and explore the search space of sequential decision making, under a reinforcement learning paradigm, precludes the inclusion of large contextualized language models, which might otherwise enable the desired generalization ability. We introduce a teacher-student imitation learning methodology and a means of converting a reinforcement learning model into a natural language understanding model. Together, these methodologies enable the introduction of contextualized language models into the sequential decision making problem space. We show that models can learn faster and generalize more, leveraging both the imitation learning and the reformulation. Our models exceed teacher performance on various held-out decision problems, by up to 7% on in-domain problems and 24% on out-of-domain problems.

## introduction we make many decisions as we interact with the world. when we are rewarded (respectively, punished), we learn to modify not only the proximal cause of the stimulus but the chain of decisions leading up to it, to encourage (respectively, discourage) future similar results. this process naturally is the paradigm of reinforcement learning (rl). policy-based learning seeks to find good estimates for q(s, a), a function that returns the expected cumulative reward (known as a q-value) if action a is chosen at state s. a desirable property of methodologies to learn q is their ability to generalize such that an appropriate action can be taken when encountering a previously unseen state. recent advances have shown strong evidence of generalization in spatiotemporal modalities such as robotic manipulation @xcite , video games @xcite , and autonomous navigation @xcite . however, in the modality of language, there is less work applying generalization approaches to decision making. useful applications of sequential decision making language models are personal assistants that proactively anticipate client needs; anti-phishing mediation agents that waste a would-be thief's time with relevant but non-helpful responses; and investigative journalist assistants that determine what to read, whom to contact, and what questions to ask to create a revelatory news report. neural reinforcement learning (rl) training approaches, such as those used to play action video games @xcite , have potential applicability in language-based decision making due to their ability to learn to navigate adversarial or exploratory scenarios. naturally, the generalization and background knowledge capability afforded by large contextualized language models such as bert @xcite may be applicable as well. a useful virtual world proxy in which to explore these approaches' applicability is that of text adventure game playing. in a text adventure game, a player is immersed in an environment by reading textual descriptions of a scene and issuing natural language commands to navigate inside the scene. the player discovers and interacts with entities and accomplishes goals, while receiving explicit rewards for doing so. learning to play text games is a useful pursuit because it is a convenient proxy for the real world cases cited above. unlike these, plentiful data for numerous games exist, an endless supply of games can be constructed, and text games have built-in re-ward functions, making them suitable for rl. this class of problems is also useful because it is challenging: after exposure to a family of games that explore the same topic and have similar gameplay (e.g., games involving cooking a specified recipe), human players perform nearly perfectly on additional games, but computer models struggle. why is this? humans quickly understand the situation they are placed in and can make rational decisions based on trial-and-error and life experience, which we can call commonsense knowledge. knowing a priori that, e.g., a closed door should be open or that it is helpful to light a lamp in a dark dungeon allows (human) players to learn faster. even though these games have the complexity of finite-state machines, computer models cannot learn to play them well. the problem appears to be due to a lack of generalization caused by a lack of commonsense. to a computer model, considering whether to fry using a fridge is no more ludicrous than considering whether to fry using a plate (which, to an untrained human cook, may be plausible, though is certainly not a good idea). both actions can be discouraged by negative reinforcement, but a human only needs to learn not to do the latter. furthermore, a computer player learning that one can chop carrots with a knife may not generalize that one can chop celery the same way, but a human surely will. there is existing work in learning to play text games with rl @xcite @xcite @xcite but the standard pattern of incorporating large language models such as bert @xcite has not yet been seen in current literature. it turns out that this integration is not trivial. most models that use bert and its ilk predominantly apply their results to supervised learning tasks that have training data with ground truth @xcite or at least, in the case of generation-based tasks like dialogue and translation, a corpus of desirable output to mimic @xcite . for tasks suited to rl such as the exploration of and interaction with a world, there is no true target or even, initially, a corpus, and thus learning can only proceed iteratively via, e.g., exploration-exploitation @xcite , which requires millions of training iterations to converge @xcite @xcite . integrating this process with the additional overhead of fine-tuning a large model like bert leads to an impractical slowdown: for the experiments considered in this work, the baseline models that use cnn require a little more than three weeks to train on an nvidia p100 gpuequipped machine. using the same models on the same tasks run for the same number of iterations on the same hardware while fine-tuning a 12-layer bert model would take more than two years. in this work, we compare different previously used representation models for deep rl through an imitation learning method that first trains a lightweight teacher using exploration-exploitation, and then uses that trained model to train a more heavyweight student model. this dramatically decreases the amount of training time needed to learn. moreover, we devise a means of casting an rl problem into a supervised learning paradigm, allowing better exploitation of large contextualized language models. in so doing, we show that agents can benefit from both the imitation learning and the reformulation, converging faster than other models, and exceeding teacher performance by 7% and 24% on both in-and out-of-domain problems, despite the limited search space. the novel contributions of this work are: the core approach of deep q-networks (dqn) as described by @xcite is to build a replay memory of partial games with associated scores, and use this to learn a function f dqn : (s, a) → r, where f dqn (s, a) is the q-value obtained by choosing action @xmath0; from s, choosing arg max ã∈a f dqn (s, ã) affords the optimal action policy and this is generally used at inference time. 2 the dqn, which predicts q given a (state, action) pair, is trained in an exploration-exploitation method known as -greedy @xcite : first, the agent plays the game stochastically, generally guided by f dqn , but with probability choosing a random action instead. the hyperparameter usually decays from 1 to 0 during the training process. as the agent plays, it collects partial play samples (s, a, r, s ), denoting taking action a at the game state s, and the immediate reward r plus the next state s reached for doing so, into a replay memory. the dqn is then improved by sampling from the replay memory, and reducing loss between f dqn and the true q, which is estimated as square error loss is minimized to improve f dqn along the gradient: the improved dqn is used to collect more replay data as the process iterates, as depicted in figure 1 (left). equation 1 shows that at every step, for every sampled state, we can only estimate loss for a single state-action pair; we do not have the r or s for actions other than a. the models eventually converge, but only after millions of training steps @xcite . ## teacher-student dqn training after running dqn training as described in section 2 for some time, our well-trained agent, which we call the teacher, can provide q-values for a set of actions a at every step. we can then collect lots of (state s, action set a, q-table q) game-play tuple data into a curriculum pool by repeatedly playing the game and obtaining f dqn -teacher (s, a) for all @xmath1. we now use that data to train a new agent f dqn -student (the student), using the same dqn approach described in section 2. however, unlike in the previous dqn scenario, the curriculum pool now contains q-values for all of a at each state. 3 we can train all actions at one step for each trajectory since we have q-values for all actions. thus the loss is and the learning procedure is as depicted on the right side of figure 1 . the main disadvantage of teacher-student learning is that in the student phase, the search space is bounded by that of the curriculum pool generated by the teacher agent. while a student model can generalize based on the curriculum pool's data, it cannot explore any more of the search space. on the other hand, student learning is much faster than teacher learning. the experience replay pool does not need to be repeatedly generated, and many more loss gradients can be calculated all at once. we will explore several architectures and configurations that take advantage of this speed. ## games and evaluation methodology unlike most video-or text-game-playing work @xcite @xcite which incrementally learns to play games through rl approaches and reports results on those same games, we evaluate on games that are not seen during learning. our games are generated from the textworld platform @xcite , which procedurally generates a wide variety of game variants with different maps, objectives, actions, threats, and background text, given user-supplied inputs. the platform provides the set of admissible actions, i.e., legal actions available at each state of each game. there are between 10-100 of these actions depending on the context. ## student experiment results having trained a teacher model using dqn and allowing unlimited exploration of the game space, we now experiment with several student learning approaches. relative to the teacher model, the students are constrained to explore using data generated from the trained teacher model. this restriction limits their ability to search but enables much faster training and, consequently, richer models. all student models are trained for 500,000 steps of 32 (trajectory, action) pairs per batch, saving checkpoints every 5,000 steps and generating results for in-and out-of-domain test sets. running on a single p100 gpu, all transformer-based models take 75-80 minutes per 5000 steps, while cnn-based models take 13 minutes. ## discussion in this section we seek to understand the following: a qualitative investigation of model performance on in-domain test sets shows that game failure arises when a model decides to prepare an ingredient improperly, (e.g., to use the bbq instead of the stove to fry). models initialized with bert have fewer such failures, indicating that bert provides background cooking knowledge, beyond what can be learned from the curriculum pool. example gameplays and complete statistics on test games are provided in the appendix. a similar pattern is observed for out-of-domain tests. one test requires the player to use four different kinds of keys with matched locked containers. as the curriculum pool does not have any information relevant for this task, models without general background knowledge suffer. in the key/lock test game (a readout is in the appendix), the teacher model repeatedly unlocks and locks a single box, and puts and takes the same key without making progress. the bert-nlu-se model, however, can correctly open the sequence of containers. figure 7 provides more insight into model performance, including an explanation for the surprising success of the cnn model. that figure shows the kl-divergence @xcite between a uniform distribution and the distribution formed from the q-values (the categorical choice distribution for bert-nlu-ce) at every step during every out-of-domain test, computed from the final point of each model. the cnn model's distribution is closer to uniform than the others. as stochastic choices are made at test time when the action distribution is uncertain (see section 4.2), the cnn model performs more exploration during the evaluation of treasure hunting games. these games do not have failure cases like the in-domain test games, so there can be some benefit to stochasticity. the other models are more confident foot_6 and, except for bert-nlu-se, are generally wrong. this result indicates that equipped with the ability to generalize from bert pre-training, bert-nlu-se has learned the skill of decision making, rather than the ability to memorize patterns. ## related work many recent works @xcite @xcite @xcite on building agents to play text-based games apply dqns @xcite or their variants. different aspects of dqn have been presented, such as action reduction with language correlation @xcite , a bounding method @xcite , the introduction of a knowledge graph @xcite ), text understanding with dependency parsing @xcite , and the bandit feedback method for agent evaluation @xcite . however, previous work uses different games to evaluate, making it difficult to compare results comprehensively. with the textworld framework's availability, there is more and more work concentrating on the generalization ability of agents, which seldom appears in the video game playing domain. @xcite work on generalization of agents on variants of a very simple coincollecting game. the simplicity of their games enables them to use an lstm-dqn method with a counting-based reward. @xcite use a knowledge graph as a persistent memory to encode states, while we use a knowledge graph later on to make actions more informative. the textworld competition has yielded a variety of works that use different approaches and methods: @xcite co-train a dqn with a question answering system for building new interactive machine reading comprehension tasks while creating agents to solve games. @xcite describe a non-rl method to learn agents, by first randomly playing on training games, then collecting all winning trajectories. by using these trajectories as training data, they manage to transform an rl problem into supervised learning. adolphs and hofmann (2020) use an actor-critic framework and prune the action-space by using hierarchical rl and a specialized module trained on a recipe database to build better agents. jain et al. (2020) apply the action elimination method proposed by @xcite on zork to the cooking games. for teacher-student training, rusu et al. ( 2015 ) design a policy distillation method that trains different agents as teacher agents. each of these teacher agents learns to play a single and separate game. then they build one student learner that can be trained with a supervised learning method to distill the policy knowledge for multi-game playing. ansari et al. ( 2018 ) also use teacher-student training for text-based games. however, our teacherstudent training method is different: we use one teacher that can play multiple games to guide multiple student agents' learning processes. ## conclusion we provide a recipe for integrating large contextualized language models and deep reinforcement learning, applying to sequential decision making and a demonstration on the proxy task of text games, showing dramatic improvements over the standard practice, particularly in out-of-domain held-out tests. we expect to apply this approach to various challenging real-world sequential decision scenarios, such as goal-directed dialogue and active information-gathering.