Generalization in Text-based Games via Hierarchical Reinforcement Learning (2021)

Abstract: Deep reinforcement learning provides a promising approach for text-based games in studying natural language communication between humans and artificial agents. However, the generalization still remains a big challenge as the agents depend critically on the complexity and variety of training tasks. In this paper, we address this problem by introducing a hierarchical framework built upon the knowledge graph-based RL agent. In the high level, a meta-policy is executed to decompose the whole game into a set of subtasks specified by textual goals, and select one of them based on the KG. Then a sub-policy in the low level is executed to conduct goal-conditioned reinforcement learning. We carry out experiments on games with various difficulty levels and show that the proposed method enjoys favorable generalizability.

## introduction text-based games are simulated systems where the agent takes textual observations as the input, and interacts with the environment via text commands @xcite . they are suitable test-beds to study natural language understanding, commonsense reasoning and language-informed decision making @xcite . reinforcement learning (rl) based agents @xcite have been developed to handle challenges such as languagebased representation learning and combinatorial action space. among them, kg-based agents (ammanabrolu and hausknecht, 2020) yield promising performance with the aid of knowledge graph (kg), which serves as a belief state to provide structural information. to design intelligent rl-based agents for textbased games, it is necessary to build agents that automatically learn to solve different games. however, generalization remains as one of the key challenges of rlthe agent tends to overfit the train-ing environment and fails to generalize to new environments @xcite . in the domain of text-based games, the textworld @xcite makes it feasible to study generalizability by creating non-overlapping game sets with customizable domain gaps (e.g., themes, vocabulary sets, difficulty levels and layouts). most previous works study generalizability either upon games with the same difficulty level but different layouts (ammanabrolu and riedl, 2019a), or upon games with a set of multiple levels that have been observed during training @xcite . although these agents perform well on relatively simple games, they can hardly achieve satisfactory performance on difficult games @xcite . in this work, we aim to develop agents that can be generalized to not only games from the same difficulty level while having unseen different layouts, but also games from unseen difficulty levels where both layouts and complexities are different. while solving a whole game might be difficult due to long-term temporal dependencies, and the learnt strategy might be difficult to be transferred to other games due to large domain gaps, it would be more flexible to treat the game as a sketch of subtasks @xcite . this brings two branches of benefits. first, the subtasks would be easier to solve as they have shortterm temporal dependencies. second, the strategies learnt for solving subtasks may be recomposed to solve an unseen game. motivated by these insights, we aim to solve a game by decomposing it into subtasks characterized by textual goals, then making decisions conditioned on them. instead of handcrafting the task sketches, we leverage the hierarchical reinforcement learning (hrl) framework for adaptive goal selection, and exploit the compositional nature of language @xcite to improve generalizability. we develop a two-level framework, hierarchical knowledge graph-based agent (h-kga) foot_0 , to learn a hierarchy of policies with the aid of kg. in the high level, we use a meta-policy to obtain a set of available goals characterized by texts, and select one of them according to the current kgbased observation. then, we use a sub-policy for goal-conditioned reinforcement learning. besides, we design a scheduled training strategy to facilitate learning across multiple levels. we conduct experiments on a series of cooking games across 8 levels, while only 4 levels are available during training. the experimental results show that our method improves generalizability in both seen and unseen levels. our contributions are summarised as follows: firstly, we are the first to study generalizability in text-based games from the aspect of hierarchical reinforcement learning. secondly, we develop a two-level hrl framework leveraging the kgbased observation for adaptive goal selection and goal-conditioned decision making. thirdly, we empirically validate the effectiveness of our method in games with both seen and unseen difficulty levels, which show favorable generalizability. ## conclusion in this paper, we investigated generalization for reinforcement learning in text-based games. we introduced a two-level hierarchical framework, h-kga, to address this problem. in the high level, a meta-policy is executed to decompose the whole game as subtasks characterized by textual goals, and select a goal based on the knowledge graphbased observation. in the low level, a sub-policy is executed to select action conditioned on the goal. experimental results showed that h-kga achieved favorable performance on games with various difficulty levels. as an ongoing work, we would like to study automatic goal generation methods. we are also interested in extending our work to more complex scenarios .