Revisiting the Roles of “Text” in Text Games (2022)

Abstract: Text games present opportunities for natural language understanding (NLU) methods to tackle reinforcement learning (RL) challenges. However, recent work has questioned the necessity of NLU by showing random text hashes could perform decently. In this paper, we pursue a fine-grained investigation into the roles of text in the face of different RL challenges, and reconcile that semantic and non-semantic language representations could be complementary rather than contrasting. Concretely, we propose a simple scheme to extract relevant contextual information into an approximate state hash as extra input for an RNN-based text agent. Such a lightweight plug-in achieves competitive performance with state-of-the-art text agents using advanced NLU techniques such as knowledge graph and passage retrieval, suggesting non-NLU methods might suffice to tackle the challenge of partial observability. However, if we remove RNN encoders and use approximate or even ground-truth state hash alone, the model performs miserably, which confirms the importance of semantic function approximation to tackle the challenge of combinatorially large observation and action spaces. Our findings and analysis provide new insights for designing better text game task setups and agents.

## introduction in text-based games @xcite @xcite , players read text observation, command text actions to interact with a simulated world, and gain rewards as they progress through the story. from a reinforcement learning (rl) viewpoint, they are partially observable markov decision processes (pomdp) -the current observation does not carry the full information of the game progress. in our figure 1 example, visiting the living room before or after the dark place puzzle may yield the same observation, but only when informed by the game history, the player can decide whether go down is the right action. recent work has proposed to incorporate rl agents with natural language understanding (nlu) capabilities for better text game performance. for example, pre-trained language models support combinatorial action generation @xcite ; commonsense reasoning @xcite , information extraction @xcite , and reading comprehension @xcite provide priors for exploration with sparse reward and long horizon; and knowledge graph @xcite and document retrieval @xcite techniques help alleviate partial observability. nevertheless, @xcite doubts the need of nlu for rl agents trained and evaluated on the same game. they found that a text game agent, drrn @xcite , performs even slightly better when rnn-based language representations are replaced with non-semantic hash codes. intuitively, hash serves to memorize state-action pairs and ignore text similarities, which is sometimes useful -consider the second-to-last observation in figure 1 and a counterfactual observation where a "lantern" is added into "inventory", rnns might encode them very similarly though they lead to antipodal consequences (die or explore the underground). how do we reconcile this with recent nlu-augmented text agents with improved performances? where are semantic representations useful, and where would a hash approach suffice? in this paper, we present initial findings that semantic and non-semantic language representations could work hand-in-hand better than each alone by targeting different rl challenges. concretely, we show the hash idea could help drrn tackle partial observability -returning to the figure 1 example, to get lantern to avoid death, it is vital to know where the lantern is, which is revealed in a previous instead of current observation. based on such intuition, we propose a simple algorithm that tracks the current location and the up-to-date descriptions of all locations, then encode them into a single approximate state hash vector as extra drrn input. though lightweight and easy-to-implement, such a representation plug-in improves drrn scores by 29% across games, with competitive performances against state-of-the-art text agents using advanced nlu techniques and pre-trained transformer models. the effectiveness is further confirmed by comparing to models that plug in groundtruth state or location hash codes, where we find our performance and these upper bounds with very little gaps. these results suggest that the current partial observability bottlenecks might not require advanced nlu models or semantic representations to conquer. however, such a message is gauged by the ablations that show the approximate state hash alone only achieves 58% of the full performance, as it fails to handle other rl challenges such as the combinatorial state and action spaces. in conclusion, we find the role of nlu in text games is not black-or-white as indicated by prior work, but rather differs for different rl challenges, and agents could benefit from combining semantic and non-semantic language representations that target different functionalities. our results and insights contribute to future research in designing better tasks and models toward autonomous agents with grounded language abilities. ## method the key to handle partial observability is to extract the appropriate state-distinguishing information from the context c t -while under-extraction leads to different states with same representations, over-extraction leads to diverging representations for the same state with different history paths. so to approximate the state hash, we first obtain and maintain a location map by exploration with limited depth d, collecting names of adjacent rooms: the algorithm details are in appendix a. ## experiments implementation details we adopt drrn hyperparameters from @xcite to train our model. following previous work, we implement the bidaf @xcite attention mechanism and the inverse dynamics auxiliary objective @xcite for better text encoding. the episodic limit is 100 steps and the training has 1,000 episodes from 8 parallel game environments. for po 1 , we use @xmath0. we train three independent runs for each game. more details are in appendix b. baselines our approach builds on the backbone drrn agent, thus we provide fair comparisons to the original drrn and its hash and inverse dynamics variants from @xcite . we also compare with more complex state-of-the-art agents that are designed to deal with the partial observability via nlu: evaluating games we select 6 games from jericho @xcite where mprc-dqn or kg-a2c exhibits performance boosts, thus are more likely to suffer from partial observability. ## discussion we propose a simple approach to deal with partial observability in text games, which could serve as a competitive baseline for future research, and also inspire similar investigations for other rl challenges to test the limits of memorization and neces-sity of nlu in different dimensions, which would in turn help identify flaws of current setups and propose better ones. we also hope our idea of best combing semantic and non-semantic language representations could be useful for building nextgeneration text game agents, as well as for other language applications with memorization needs like closed-domain qa or goal-oriented dialog.