Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents (2021)

Abstract: Text-based games simulate worlds and interact with players using natural language. Recent work has used them as a testbed for autonomous language-understanding agents, with the motivation being that understanding the meanings of words or semantics is a key component of how humans understand, reason, and act in these worlds. However, it remains unclear to what extent artificial agents utilize semantic understanding of the text. To this end, we perform experiments to systematically reduce the amount of semantic information available to a learning agent. Surprisingly, we find that an agent is capable of achieving high scores even in the complete absence of language semantics, indicating that the currently popular experimental setup and models may be poorly designed to understand and leverage game texts. To remedy this deficiency, we propose an inverse dynamics decoder to regularize the representation space and encourage exploration, which shows improved performance on several games including Zork I. We discuss the implications of our findings for designing future agents with stronger semantic understanding.

## introduction text adventure games such as zork i (figure 1 (a)) have been a testbed for developing autonomous agents that operate using natural language. since interactions in these games (input observations, action commands) are through text, the ability to understand and use language is deemed necessary and critical to progress through such games. previous work has deployed a spectrum of methods for language processing in this domain, including word vectors @xcite , recurrent neural networks @xcite , pre-trained language models @xcite , open-domain question answering systems @xcite , knowledge graphs @xcite @xcite , and reading comprehension systems @xcite . meanwhile, most of these models operate under the reinforcement learning (rl) framework, where the agent explores the same environment in repeated episodes, learning a value function or policy to maximize game score. from this perspective, text games are just special instances of a partially observable markov decision process (pomdp) (s, t, a, o, r, γ), where players issue text actions @xmath0, receive text observations @xmath1, a), and the underlying game state @xmath2, a). however, what distinguishes these games from other pomdps is the fact that the actions and observations are in language space l. therefore, a certain level of decipherable semantics is attached to text observations @xmath3. ideally, these texts not only serve as observation or action identifiers, but also provide clues about the latent transition function t and reward function r. for example, issuing an action "jump" based on an observation "on the cliff" would likely yield a subsequent observation such as "you are killed" along with a negative reward. human players often rely on their understanding of language semantics to inform their choices, even on games they have never played before, while replacing texts with non-semantic identifiers such as their corresponding hashes (figure 1 (c )) would likely render games unplayable for people. however, would this type of transformation affect current rl agents for such games? in this paper, we ask the following question: to what extent do current reinforcement learning agents leverage semantics in text-based games? to shed light on this question, we investi-(a) zork i observation 21: you are in the living room. there is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. you are carrying: a brass lantern . . . ## results setup we train on 12 games † from the jericho benchmark @xcite . these human-written interactive fictions are rich, complex, and diverse in semantics. ‡ for each game, we train drrn asynchronously on 8 parallel instances of the game environment for 10 5 steps, using a prioritized replay buffer. following prior practice @xcite , we augment observations with location and inventory descriptions by issuing the 'look' and 'inventory' commands. we train three independent runs for each game and report their average score. for hash, we use the python built-in hash function to process text as a tuple of token ids, and implement the random vector generator g by seeding pytorch with the hash value. for inv-dy, we use λ 1 = λ 2 = 1. transfer we also investigate if representations of different models can transfer to a new language environment, which is a potential benefit of learning natural language semantics. so we consider the two most similar games in jericho, zork i and zork iii, fix the language encoders of different zork i models, and re-train the q-network on zork iii for 10,000 steps. as shown in figure 2 , inv-dy representations can achieve a score around 1, which surpasses the best result of models trained from scratch on zork iii for 100,000 steps (around 0.4), showing great promise in better gameplay by leveraging language understanding from other games. hash transfer is equivalent to training from scratch as the representations are not learnt, and a score around 0.3 is achieved. finally, drrn representations transfer worse than hash, possibly due to overfitting to the td loss (2). visualizations finally, we use t-sne @xcite to visualize representations of some zork i walkthrough states in figure 3 . the first 30 walkthrough states (red, score 0-45) are well experienced by the models during exploration, whereas the last 170 states (blue, score 157-350) are unseen § . we also encircle the subset of states at location 'living room' for their shared semantics. first, we note that the hash representations for living room states are scattered randomly, unlike the other two models with gru language encoders. further, the base drrn overfits to the td loss (2), representing unseen states (blue) in a different subspace to seen states (red) without regarding their semantic similarity. ind-dy is able to extrapolate to unseen states and represent them similarly to seen states for their shared semantics, which may explain its better performance on this game. game stochasticity all the above experiments were performed using a fixed game random seed for each game, following prior work @xcite . to investigate if randomness in games affects our conclusions, we run one trial of each game with episode-varying random seeds ¶ . we find the average normalized score for the base drrn, hash, inv-dy to be all around 17%, with performance drop mainly on three stochastic games (dragon, zork i, zork iii). notably, the core finding that the base drrn and hash perform similarly still holds. intuitively, even though the q-values would be lower overall with unexpected transitions, rl would still memorize observations and actions that lead to high q-values. ## discussion at a high level, rl agents for text-based games succeed by (1) exploring trajectories that lead to high scores, and (2) learning representations to stably reach high scores. our experiments show that a semantics-regularized inv-dy model manages to explore higher scores on some games (dragon, omniquest, zork i), while the hash model manages to memorize scores better on other games (library, ludicorp, pentari) using just a fixed, random, non-semantic representation. this leads us to hypothesize two things. first, fixed, stable representations might make q-learning easier. second, it might be desirable to represent similar texts very differently for better gameplay, e.g. the q-value can be much higher when a key object is mentioned, even if it only adds a few words to a long observation text. this motivates future thought into the structural vs. functional use of language semantics in these games. our findings also urge a re-thinking of the popular 'rl + valid action handicap' setup for these games. on one hand, rl sets training and evaluation in the same environment, with limited text corpora, and sparse, mostly deterministic rewards as the only optimization objective. such a combination easily results in overfitting to the reward system of a specific game (figure 2 ), or even just a specific stage of the game (figure 3 ). on the other hand, the valid action handicap reduces the action set to a small size tractable for memorization, and reduces the language understanding challenge for the rl agent. thus for future research on text-based games, we advocate for more attention towards alternative setups without rl or handicaps @xcite @xcite . particularly, in a 'rl + no valid action handicap' setting, generating action candidates rather than simply choosing from a set entails more opportunities and challenges with respect to learning grounded language semantics @xcite . additionally, training agents on a distribution of games and evaluating them on a separate set of unseen games would require more general semantic understanding. semantic evaluation of these proposed paradigms is outside the scope of this paper, but we hope it will spark a productive discussion on the next steps toward building agents with stronger semantic understanding.