Self-imitation Learning for Action Generation in Text-based Games (2023)

Abstract: In this work, we study reinforcement learning (RL) in solving text-based games. We address the challenge of combinatorial action space, by proposing a confidence-based self-imitation model to generate action candidates for the RL agent. Firstly, we leverage the self-imitation learning to rank and exploit past valuable trajectories to adapt a pre-trained language model (LM) towards a target game. Then, we devise a confidence-based strategy to measure the LM’s confidence with respect to a state, thus adaptively pruning the generated actions to yield a more compact set of action candidates. In multiple challenging games, our model demonstrates promising performance in comparison to the baselines.

## introduction text-based games are situated systems where the game agents observe textual descriptions, and generate textual commands to interact with the environment. these games have proven to be suitable test-beds for studying various natural language processing (nlp) tasks, such as question answering @xcite , dialogue systems @xcite , situated language learning @xcite and commonsense reasoning @xcite . recent years have witnessed the thriveness of designing reinforcement learning (rl) agents in solving these games @xcite @xcite , while the combinatorial action space remains as a challenging issue, preventing rl agents from being deployed in real world applications. in general, text-based games accept free-form actions, resulting in a large combinatorial action space. fig. 1 shows a raw excerpt from the classic game "zork1". a 4-word action has to be selected from |v| 4 candidates, where v denotes the vocabulary set @xcite . given that only 130 actions are required to solve this game, the agent wastes both training data and time in attempting irrelevant actions @xcite . to handle the combinatorial action space, early efforts either heavily rely on hand-crafted rules, or simply assume the availability of the action candidate set. for example, some works consider a set of currently admissible actions @xcite , or a template-based action space @xcite . alternatively, some other works alleviated this challenge by filtering inadmissible actions through methods such as action affordance @xcite , bandit-based elimination @xcite and rule-based scoring @xcite . in handling the combinatorial action space for text-based games, recent pre-trained language models (plms) @xcite @xcite can help generate actions. however, the potential of lm is still less effectively explored. as one of the pioneer works, @xcite proposed the calm, which is a gpt-2 model pre-trained on human gameplay trajectories, to generate the action candidate set for the rl agent. however, when solving a previously unseen game, calm tends to generate actions with less satisfying qualities, leading to two consequences that may affect rl training: 1) the action set may contain a large proportion of inadmissible actions, and 2) the useful actions may not be generated. as a mitigation, the calm model is set to generate a relatively huge action candidate set, followed by ad-hoc operations to filter out the inadmissible actions, which requires prior knowledge. @xcite extended the lm-based agent to goal-conditioned tasks to follow instructions. besides the offline pretraining data, the lm is further improved with the successful trajectories collected during online interaction. however, text-based games do not have well-defined goals. furthermore, some games are so challenging that it is impossible to collect successful trajectories @xcite . in this work, we address the crux of combinatorial action space in solving text-based games. we propose the confidence-based self-imitation model (csm) to generate the action candidates for the rl agent. * firstly, we leverage the selfimitation learning method @xcite to rank and exploit past trajectories of high values to adapt a pre-trained lm towards the target game. then, we propose a confidence-based strategy to measure the lm's confidence @xcite with respect to a state, thus adaptively pruning the action candidates based on the confidence value. our model achieves promising performance in six challenging man-made games. apart from significantly outperforming an action generation-based baseline, our strategy helps the rl agent to even achieve comparable performance to a baseline armed with the oracle action candidate set. our main contributions are summarized as follows: firstly, we develop a lm-based framework to handle the issue of combinatorial action space in solving text-based games. secondly, we pro-* code is available at https://github.com/winni18/csm . pose a strategy to further improve the lm via selfimitation learning during the rl training. thirdly, our experiments demonstrate that, the proposed method significantly improve the performance on multiple games compared with the strong contemporary method. ## conclusion in this work, we studied reinforcement learning in solving the text-based game. we proposed the csm framework to generate a set of action candidates for the rl agent, which alleviates the issue of combinatorial action space. during rl training, we collected and exploited past high-quality trajectories and utilised self-imitation learning to improve the language model. in addition, a confidencebased action pruning strategy was proposed to further restrict the action space. we evaluate our method using the jericho benchmark. in a variety of text-based games, our method significantly improves the performance compared with the strong contemporary method, and even overcomes the challenging bottleneck in the game "snacktime".