Fire Burns, Sword Cuts: Commonsense Inductive Bias for Exploration in Text-based Games (2022)

Abstract: Text-based games (TGs) are exciting testbeds for developing deep reinforcement learning techniques due to their partially observed environments and large action spaces. In these games, the agent learns to explore the environment via natural language interactions with the game simulator. A fundamental challenge in TGs is the efficient exploration of the large action space when the agent has not yet acquired enough knowledge about the environment. We propose CommExpl, an exploration technique that injects external commonsense knowledge, via a pretrained language model (LM), into the agent during training when the agent is the most uncertain about its next action. Our method exhibits improvement on the collected game scores during the training in four out of nine games from Jericho. Additionally, the produced trajectory of actions exhibit lower perplexity, when tested with a pretrained LM, indicating better closeness to human language.

## introduction text-based games (tgs) are environments where agents learn to comprehend situations in language and produce decisions in language @xcite @xcite . deep reinforcement learning lends itself as a natural paradigm to solve tgs due to its ability to learn from unsupervised game playing experience. however, existing rl agents are far away from solving tgs due to their combinatorially large action spaces that hinders efficient exploration @xcite . @xcite ; @xcite proposed incorporating a belief knowledge graph (bkg) built from the textual observations to help the agent reason more effectively about observed objects during the gameplay. most of the recent works neglected linguistic aspects of tgs and focused on the construction and utilisation of bkg @xcite @xcite @xcite . some exceptions involve developing pre-trained language models (lms) to propose action candidates for a given observation @xcite , and investigating the relationship between semantic coherence and state representations @xcite . in parallel, it has been argued that recent pretrained lms capture commonsense factual knowledge about the world @xcite @xcite . more direct attempt in this direction was the commonsense transformer (comet) which is a lm fine-tuned explicitly on commonsense knowledge graph (cskg), to explicitly generate commonsense inferences @xcite . prior works with commonsense focused on completing bkg using pre-defined cskg @xcite or dynamic comet-generated commonsense inferences @xcite . nonetheless, there is no work on explicitly using commonsense as an inductive bias in the context of exploration for tgs. to bridge the gap, we propose commonsense exploration (commexpl) which constructs a cskg dynamically, using comet, based on the state of textual observation per step. then, the natural language actions are scored with comet and agent, to re-rank the policy distributions. we refer to this as applying commonsense conditioning. however, doing this throughout the whole training is expensive and may not be beneficial as gameplay is not led by commonsense. to rectify this, we propose an entropy scheduler, driven by the entropy of the policy distribution, to regulate applying commonsense conditioning. ## approach notations. text-based games are modelled as a partially observable markov decision processes (pomdps) of a tuple of ⟨s, a, p, o, ω, r, γ⟩, where s, a, ω denote sets of states, actions, and observations, respectively. also, r and γ denote the reward function and the discount factor, while p and o denote the transition probabilities and set of conditional observations probabilities, respectively. the agent requires to map an observation to a state (ω → s) and produce a policy π. by selecting an action a t from the policy π, the agent changes current state s t , receives a reward signal r, receives an observation through transition p(s t+1 |s t , a t ), and also receives a conditional observation o(ω t |s t ). the agent learns the policy π θ (a|o) that maximizes the expectation of the cu-mulative reward function e ∞ @xmath0, a t ) . ## conclusion we investigated the effect of commonsense in textbased rl agent during the training. our results show that despite the hyper-parameters tuning on a single game, the proposed approach improves on other gaming environments in jericho, total four out of nine. furthermore, injecting commonsense also positively influences the semantics of natural language actions, resulting in lower perplexity. our future work will extend its application to different text-based environments and investigate how this linguistic properties from lm helps the agent.