Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020)

Abstract: Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. In this paper, we propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. Our key insight is to train language models on human gameplay, where people demonstrate linguistic priors and a general game sense for promising actions conditioned on game history. We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize in-game rewards. We evaluate our approach using the Jericho benchmark, on games unseen by CALM during training. Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model. Surprisingly, on half of these games, CALM is competitive with or better than other models that have access to ground truth admissible actions. Code and data are available at https://github.com/princeton-nlp/calm-textgame .

## introduction text-based games have proven to be useful testbeds for developing agents that operate in language. as interactions in these games (input observations, action commands) are through text, they require solid language understanding for successful gameplay. while several reinforcement learning (rl) models have been proposed recently @xcite @xcite , combinatorially large action spaces continue to make these games challenging for these approaches. the action space problem is exacerbated by the fact that only a tiny fraction of action commands are admissible in any given game state. an admissible action is one that is parseable by the game observation: you are in the living room. there is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. you are carrying: a brass lantern . . . ## experimental setup we perform empirical studies to 1) evaluate the quality of actions generated by calm in isolation from the complexities of rl, 2) evaluate calm combined with an rl agent for gameplay, and 3) analyze what factors contribute to the effectiveness of our method. we describe our setup in this section and provide results in section 5. ## conclusion in this paper, we proposed the contextual action language model (calm), a language model approach to generating action candidates for reinforcement learning agents in text-based games. our key insight is to use language models to capture linguistic priors and game sense from humans gameplay on a diverse set of games. we demonstrated that calm can generate high-quality, contextuallyrelevant actions even for games unseen in its training set, and when paired with a drrn agent, outperforms previous approaches on the jericho benchmark @xcite by as much as 69% in terms of average normalized score. remarkably, on many of these games, our approach is competitive even with models that use ground truth admissible actions, implying that calm is able to generate high-quality actions across diverse games and contexts. from the results in vishal jain, william fedus, hugo larochelle, doina precup, and marc g. bellemare. 2019. algorithmic improvements for deep reinforcement learning applied to interactive fiction. corr, abs/1903.03094. armand joulin, edouard grave, piotr bojanowski, and tomas mikolov. 2017. bag of tricks for efficient text classification. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pages 427-431. association for computational linguistics. daniel jurafsky and james h. martin. 2009. speech and language processing (2nd edition). prentice-hall, inc., usa. bartosz kostka, jaroslaw kwiecien, jakub kowalski, and pawel rychlikowski. 2017. text-based adventures of the golovin ai agent. corr, abs/1705.05637. angeliki lazaridou, anna potapenko, and olivier tieleman. 2020. multi-agent communication meets natural language: synergies between functional and structural language learning. arxiv preprint arxiv:2005.07064. tomas mikolov, ilya sutskever, kai chen, greg s corrado, and jeff dean. 2013. distributed representations of words and phrases and their compositionality. in c. j. c. burges, l. bottou, m. welling, z. ghahramani, and k. q. weinberger, editors, advances in neural information processing systems 26, pages 3111-3119. curran associates, inc. karthik narasimhan, tejas d. kulkarni, and regina barzilay. 2015. language understanding for textbased games using deep reinforcement learning. in emnlp, pages 1-11. alec radford, jeffrey wu, rewon child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. openai blog, 1(8):9. yiping song, rui yan, xiang li, dongyan zhao, and ming zhang. 2016. two are better than one: an ensemble of retrieval-and generation-based dialog systems. arxiv preprint arxiv:1610.07149. ilya sutskever, oriol vinyals, and quoc v le. 2014. sequence to sequence learning with neural networks. in advances in neural information processing systems, pages 3104-3112. ruo yu tao, marc-alexandre côté, xingdi yuan, and layla el asri. 2018. towards solving text-based games by producing adaptive action spaces. arxiv preprint arxiv:1812.00855. jack urbanek, angela fan, siddharth karamcheti, saachi jain, samuel humeau, emily dinan, tim rocktäschel, douwe kiela, arthur szlam, and jason weston. 2019. learning to speak and act in a fantasy text adventure game. corr, abs/1903.03094. nick walton. 2019. ai dungeon 2: creating infinitely generated text adventures with deep learning language models. jason d. williams, kavosh asadi, and geoffrey zweig. 2017. hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 665-677, vancouver, canada. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. tom zahavy, matan haroush, nadav merlis, daniel j mankowitz, and shie mannor. 2018. learn what not to learn: action elimination with deep reinforcement learning. in s. bengio, h. wallach, h. larochelle, k. grauman, n. cesa-bianchi, and r. garnett, editors, advances in neural information processing systems 31, pages 3562-3573. curran associates, inc. tiancheng zhao and maxine eskenazi. 2016. towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. in proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 1-10, los angeles. association for computational linguistics.