A Survey of Text Games for Reinforcement Learning Informed by Natural Language (2022)

Abstract: Reinforcement Learning has shown success in a number of complex virtual environments. However, many challenges still exist towards solving problems with natural language as a core component. Interactive Fiction Games (or Text Games) are one such problem type that offer a set of safe, partially observable environments where natural language is required as part of the Reinforcement Learning solution. Therefore, this survey’s aim is to assist in the development of new Text Game problem settings and solutions for Reinforcement Learning informed by natural language. Specifically, this survey: 1) introduces the challenges in Text Game Reinforcement Learning problems, 2) outlines the generation tools for rendering Text Games and the subsequent environments generated, and 3) compares the agent architectures currently applied to provide a systematic review of benchmark methodologies and opportunities for future researchers.

## introduction language is often used by humans to abstract, transfer, and communicate knowledge of their decision making when completing complex tasks. however, traditional reinforcement learning (rl) methods, such as the prominent neural agents introduced by @xcite and @xcite , are limited to single task environments defined and solved without any language. @xcite specified that further studies are required for improving solutions on problems that necessitate the use of language. the authors also note that language may improve solutions on problems that are traditionally solved without language. this includes the use of neural agents pre-trained on natural language corpora transferring syntactic and semantic information to future tasks. an example is the use of the similarities between the tasks of chopping and cutting a carrot in cookingworld @xcite -that is, completing one should allow you to transfer the decision making to the other. to aid in developing solutions on these problems, we pose text games as a testing environment as they simulate complex natural language problems in controllable settings. in other words, researchers can generate the text games with limitations to evaluate any number of the specific challenges given by @xcite and @xcite . detailed descriptions of the possible controls (defined as 'handicaps') are provided in section 2.4. likewise, the challenges with currently posed solutions are provided in section 2.3 and are summarized by the following: text games are also safe in that we are not making decisions on a 'live' system (i.e., one that affects the real-world). this allows agents to explore freely without limitations and is often a requirement of training any reinforcement learning agent. furthermore, this has also been used to evaluate methods with unique goals such as @xcite , who wanted to maximize exploration to new states and achieved this by adding a reward to the first encounter of an unseen state. so far, research has mostly been performed independently, with many authors generating their own environments to evaluate their proposed architectures. this lack of uniformity in the environments makes comparisons between authors challenging and a need for structuring recent work is essential for systematic comparability. formally, this survey provides the first systematic review of the challenges posed by the generation tools designed for text game evaluation. furthermore, we also provide details on the currently generated environments and the resultant rl agent architectures used to solve them. this acts as a complement to prior studies in the challenges of real-world rl and rl informed by natural language by @xcite and @xcite , respectively. ## text games and reinforcement learning text games are turn-based games that interpret and execute player commands to update the current position within the game environment. the current state is provided to the player in language and the player must take actions by entering textual commands that are parsed and confirmed to be valid or not. many of these games were designed by human developers for human players based on real-world logic with a clear definition of how a player may win the game (e.g., zork @xcite ). in many cases, common knowledge is used to complete the task efficiently as, for example, a key encountered at one point will be needed to progress through locked doors later in the game. this creates a complex set of decisions that are connected by contextual knowledge of how objects interact with each other to reach a clear goal. reinforcement learning is a natural solution, as the problem can be defined by a sequence of states dependent on the player's actions with a reward signal defined on the game's win condition. however, an agent will also need to find an efficient solution to the linguistic challenges that a human player would experience when exploring through the text-based game. in this section, we first formalize the reinforcement learning model before introducing the text game generators, challenges, and possible controls to provide the essential background to the published methods introduced in later sections. ## benchmark environments and agents thus far, the majority of researchers have independently generated their own environments with textworld @xcite . as the field moves towards more uniformity in evaluation, a clear overview of which environments are already generated, their design goals and the benchmark approach is needed. we find that many of the environments focus on exploring increasingly complex environments to 'collect' an item of some kind ranging from a simple coin to household objects. this is due to the textworld generator's well defined logic rules for such objects but can also be a limitation on the possible scope of the environment. when evaluating an agent's performance, three types of evaluation settings are typically considered: the difficulty settings of the environments are defined by the complexity of the challenges that the agents are required to overcome. in most cases, these have been limited to just a few levels. however, cookingworld defines the challenge by a set of key variables that can be changed separately or in combination and therefore does not offer clear discrete difficulty settings. the max number of rooms and objects depends heavily on the type of task. coin collecter, for example, has only 1 object to find as the complexity comes from traversing a number of rooms. alternatively, cookingworld has a limited number name encoder action selector kg ptf pre-training as tasks ammanabrolu et al. (2020a) gru a2c dl albert j-qa tb zork1 xu et al. (2020) gru a2c dl none none tb jsog ammanabrolu and hausknecht (2020) gru a2c dl none clubfloyd tb jsog murugesan et al. (2020b) gru a2c dl+cs none glove cb tw-commonsense yao et al. (2020) gru drrn none gpt-2 clubfloyd cb jsog adolphs and hofmann (2020) bi-gru a2c none none ts, glove 100 tb cw guo et al. (2020) bi-gru dqn none none glove 100 tb jsog xu et al. (2020) tf drrn none none none tb jsog yin and may (2020) tf,lstm dsqn none none none ns cw, th adhikari et al. (2020) r-gcn, tf ddqn dl none ts cb tw-cook zahavy et al. (2018) cnn dqn none none word2vec 300 cb zork1 ammanabrolu and riedl (2019) lstm dqn dl none ts, glove 100 cb tw-home he et al. (2016) bow drrn none none none cb other narasimhan et al. (2015) lstm dqn none none none pb other yin et al. (2020) bert dqn dl bert none cb ftwp, th madotto et al. (2020) lstm seq2seq none none glove 100,300 pb cc, cw ## conclusion many of the challenges faced within text games also exist in other language-based reinforcement learning problems, highlighted by @xcite . likewise, these challenges also exist in 'real-world' settings noted by @xcite including: limited data, safety constraints, and/or a need for interpretability. we find that most text games methodologies focus on either an agent's ability to learn efficiently or generalize knowledge. a continued development in this direction is the primary motive for much of the recent research and could lead to solutions that work sufficiently well on these real-world problem settings for the primary challenge of limited and/or expensive data. likewise, specific work on interpretability (such as that of @xcite and language acquisition may transfer to problems that interact in the real-world directly with humans. to this end, this study has summarized recent research developments for interactive fiction games that have generated a set of environments and tested their architectures for generalization and overfitting capabilities. with the comparisons made in this work and a uniform set of environments and baselines, new architectures can be developed and then systematically evaluated for improving results within text games and subsequently other reinforcement learning problems that include language.