Perceiving the World: Question-guided Reinforcement Learning for Text-based Games (2022)

Abstract: Text-based games provide an interactive way to study natural language processing. While deep reinforcement learning has shown effectiveness in developing the game playing agent, the low sample efficiency and the large action space remain to be the two major challenges that hinder the DRL from being applied in the real world. In this paper, we address the challenges by introducing world-perceiving modules, which automatically decompose tasks and prune actions by answering questions about the environment. We then propose a two-phase training framework to decouple language learning from reinforcement learning, which further improves the sample efficiency. The experimental results show that the proposed method significantly improves the performance and sample efficiency. Besides, it shows robustness against compound error and limited pre-training data.

## introduction text-based games are simulated environments where the player observes textual descriptions, and acts using text commands @xcite . these games provide a safe and interactive way to study natural language understanding, commonsense reasoning, and dialogue systems. besides language processing techniques, reinforcement learning has become a quintessential methodology for solving text-based games. some rl-based game agents have been developed recently and proven to be effective in handling challenges such as language representation learning and partial observability @xcite @xcite . despite the effectiveness, there are two major challenges for rl-based agents, preventing them from being deployed in real world applications: the low sample efficiency, and the large action space @xcite . the low sample efficiency is a crucial limitation of rl which refers to the fact that it typically requires a huge amount of data to train an agent to achieve human-level performance @xcite . this is because human beings are usually armed with prior knowledge so that they don't have to learn from scratch @xcite . in a language-informed rl system, in contrast, the agent is required to conduct both language learning and decision making regimes, where the former can be considered as prior knowledge and is much slower than the later @xcite . the sample efficiency could be improved through pre-training methods, which decouple the language learning from decision making @xcite . the selection of pre-training methods thus plays an important role: if the pre-trained modules perform poorly on unseen data during rl training, the incurred compound error will severely affect the decision making process. another challenge is the large discrete action space: the agent may waste both time and training data if attempting irrelevant or inferior actions @xcite . in this paper, we aim to address these two challenges for reinforcement learning in solving textbased games. since it is inefficient to train an agent to solve complicated tasks (games) from scratch, we consider decomposing a task into a sequence of subtasks as inspired by @xcite . we design an rl agent that is capable of automatic task decomposition and subtask-conditioned action pruning, which brings two branches of benefits. first, the subtasks are easier to solve, as the involved temporal dependencies are usually shortterm. second, by acquiring the skills to solve subtasks, the agent will be able to learn to solve a new task more quickly by reusing the learnt skills @xcite . the challenge of large action space can also be alleviated, if we can filter out the actions that are irrelevant to the current subtask. inspired by the observation that human beings can understand the environment conditions through question answering @xcite , we design worldperceiving modules to realize the aforementioned functionalities (i.e., task decomposition and action pruning) and name our method as question-guided world-perceiving agent (qwa) * . fig. 1 (b) shows an example of our decision making process. being guided by some questions, the agent first decomposes the task to obtain a set of available subtasks, and selects one from them. next, conditioned on the selected subtask, the agent conducts action pruning to obtain a refined set of actions. in order to decouple language learning from decision making, which further improves the sample efficiency, we propose to acquire the world-perceiving modules through supervised pre-training. we design a two-phase framework to train our agent. in the first phase, a dataset is built for the training of the world-perceiving modules. in the second phase, we deploy the agent in games with the pretrained modules frozen, and train the agent through reinforcement learning. we conduct experiments on a series of cooking games. we divide the games as simple games and complex games, and construct the pre-training dataset from simple games only. the experimental results show that qwa achieves high sample efficiency in solving complex games. we also show that our method enjoys robustness against compound error and limited pre-training data. our contributions are summarized as follows: firstly, we develop an rl agent featured with question-guided task decomposition and action space reduction. secondly, we design a two-phase framework to efficiently train the agent with limited data. thirdly, we empirically validate our method's effectiveness and robustness in complex games. ## conclusion in this paper, we addressed the challenges of low sample efficiency and large action space for deep reinforcement learning in solving text-based games. we introduced the world-perceiving modules, which are capable of automatic task decomposition and action pruning through answering questions about the environment. we proposed a twophase training framework, which decouples the language learning from the reinforcement learning. experimental results show that our method achieves improved performance with high sample efficiency. besides, it shows robustness against compound error and limited pre-training data. regarding the future work, we would like to further improve the pre-training performance by introducing contrastive learning objective @xcite and kg-based data augmentation @xcite .