Bootstrapped Q -learning with Context Relevant Observation Pruning to Generalize in Text-based Games (2020)

Abstract: We show that Reinforcement Learning (RL) methods for solving Text-Based Games (TBGs) often fail to generalize on unseen games, especially in small data regimes. To address this issue, we propose Context Relevant Episodic State Truncation (CREST) for irrelevant token removal in observation text for improved generalization. Our method first trains a base model using Q-learning, which typically overfits the training games. The base model’s action token distribution is used to perform observation pruning that removes irrelevant tokens. A second bootstrapped model is then retrained on the pruned observation text. Our bootstrapped agent shows improved generalization in solving unseen TextWorld games, using 10x-20x fewer training games compared to previous state-of-the-art (SOTA) methods despite requiring fewer number of training episodes.

## introduction reinforcement learning (rl) methods are increasingly being used for solving sequential decisionmaking problems from natural language inputs, like text-based games @xcite @xcite @xcite and personal conversation assistants @xcite @xcite . in this work, we focus on text-based games (tbgs), which require solving goals like "obtain coin from the kitchen", based on a natural language description of the agent's observation of the environment. to interact with the environment, the agent issues text-based action commands ("go west") upon which it receives a reward signal used for training the rl agent. tbgs serve as testbeds for interactive real-world tasks goal: who's got a virtual machine and is about to play through an fast paced round of textworld? you do! retrieve the coin in the balmy kitchen. observation: you've entered a studio. you try to gain information on your surroundings by using a technique you call "looking." you need an unguarded exit ? you should try going east. you need an unguarded exit? you should try going south. you don't like doors? why not try going west, that entranceway is unblocked. ## experimental results setup: we used easy, medium, and hard modes of the coin-collector textworld @xcite framework for evaluating our model's generalization ability. the agent has to collect a coin that is located in a particular room. we trained each method on various numbers of training games (denoted by n#) to evaluate generalization ability from a few numbers of games. ## conclusion we present a method for improving generalization in tbgs by removing irrelevant tokens from observation texts. our bootstrapped model -trained on the salient observation tokens -obtains generalization performance similar to sota methodswith 10x-20x fewer training games -due to better generalization; and shows accelerated convergence. in this paper, we have restricted our analysis to tbgs that feature similar domain distributions in train and test games. in the future, we will focus our attention on the topic of generalization in the presence of domain differences such as novel objects; and given goal statements in test games that were not seen by the agent during training.