Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games (2023)

Abstract: Text-based games present an exciting test-bed for reinforcement learning algorithms in the natural language environment. In these adventure games, an agent must learn to interact with the environment through text in order to accomplish tasks, facing large and combinational action space as well as partial observability issues. However, existing solutions fail to decompose the task and abstract the action autonomously, which either pre-specify the subtasks or pre-train on the human gameplay dataset. In this work, we introduce a novel skill-centric reinforcement learning framework, which is capable of abstracting the action in an end-to-end manner. To learn a more disentangled skill, we focus on the informativeness and distinguishability of the skill in accordance with the information bottleneck principle. Specifically, we introduce a discriminator to enable the skill to reflect the trajectory and push their representations onto the unit hypersphere to distribute uniformly. Moreover, a self-predictive mechanism is employed to learn inverse and forward dynamics, and a self-recovery mechanism is leveraged to refine the action representation, thus resulting in a more comprehensive perception of dynamics and more effective representations of textual state and action. Empirical experiments are carried out on the Jericho environment and the results validate the superiority against state-of-the-art baselines.

## introduction mastering the ability of understanding and responding using natural language is essential for a wide range of technologies and applications (e.g., in customer consultation and service systems). the interactive adventure games @xcite , such as zork1 (can be seen in action abstraction, a.k.a. "skill", refers to a set of pertinent long-horizon behaviors that an agent is capable of engaging in, which is a temporally extended macro-action in hierarchical reinforcement learning. for instance, there are several steps involved in entering a house, such as finding the door, approaching it and opening the door. leveraging such action abstraction can assist the intelligent agent in autonomously breaking down the challenging task into a hierarchy, where the action space issue becomes trivial. motivated by this, to cope with the large action space challenge, we choose to abstract the action from a novel perspective, without requiring the assistance of any pretrained models or prior knowledge. additionally, observing the textual state and action in the environment, the traditional representation extractor designed for vector-based state and action is insufficient, necessitating the mining and exploitation of comprehensive information. thus, it is desirable to employ a better representation learner for textual state and action. with valuable semantic representation acquired from those textual observations, intelligent agents deployed on top of them will bring more effective use of them naturally. in this work, we introduce a skill-centric action abstraction framework in an end-to-end manner for text-based games, in the hopes of alleviating the combinatorially-size action space and partial observability issues. a novel skill learning strategy is designed according to the information bottleneck principle, with the goal of a disentangled skill by enhancing the informativeness and distinguishability of the skill. in particular, in pursuing informativeness, we pull the representations of skill and the current trajectory closely by maximizing the mutual information between them, and push the representations of skill and the other unrelated trajectory away by minimizing their mutual information. by this means, the skill is enforced to cover sufficient information about the current trajectory while being parsimonious to exclude unrelated noisy information. in pursuing distinguishability, the representations of skill and trajectory are separated onto the unit hypersphere, thus preserving maximal information. the discovered skill can guide the agent to make optimal decisions. besides, within the skill-centric framework, the self-predictive mechanisms for inverse and forward dynamics and the self-recovery dynamics learning for the action itself are presented. the core idea lies in easing the partial information issue and reinforcing the representations of textual state and action effectively. overall, our contributions can be summarized as follows: ## methodology figure 1 depicts the whole process of our proposed framework. besides the general reinforcement learning framework with agent and environment, we introduce the skill and textual state and action representation learning in text-based games, which focus on large action space and partial observability issues, and aid the agent in making the optimal decision. ## conclusion in this work, we present a novel skill-centric reinforcement learning framework for text-based games. our framework aims at abstracting the action in an end-to-end manner, in line with the information bottleneck principle which pursues the informativeness and distinguishability of skill. for better perceiving textual state and action, we employ self-predictive mechanisms for forward and inverse dynamics and a self-recovery mechanism for the action itself. to the best of our knowledge, this is the first work to consider skill discovery and representation learning in text-based games aiming at eliminating partial observability and large action space issues, without the need of any pre-trained models or prior knowledge.