Efficient Text-based Reinforcement Learning by Jointly Leveraging State and Commonsense Graph Representations (2021)

Abstract: Text-based games (TBGs) have emerged as useful benchmarks for evaluating progress at the intersection of grounded language understanding and reinforcement learning (RL). Recent work has proposed the use of external knowledge to improve the efficiency of RL agents for TBGs. In this paper, we posit that to act efficiently in TBGs, an agent must be able to track the state of the game while retrieving and using relevant commonsense knowledge. Thus, we propose an agent for TBGs that induces a graph representation of the game state and jointly grounds it with a graph of commonsense knowledge from ConceptNet. This combination is achieved through bidirectional knowledge graph attention between the two symbolic representations. We show that agents that incorporate commonsense into the game state graph outperform baseline agents.

## introduction text-based games (tbgs) are simulation environments in which an agent interacts with the world purely in the modality of text. tbgs have emerged as key benchmarks for studying how reinforcement learning agents can tackle the challenges of language understanding, partial observability, and action generation in combinatorially large action spaces. one particular text-based gaming environment, textworld @xcite , has received significant attention in recent years. recent work has shown the need for additional knowledge to tackle the challenges in tbgs. @xcite proposed handcrafted rules to represent the current state of the game using a state knowledge graph (much like a map of the game). our own prior work @xcite proposed an extension of textworld, called textworld commonsense (twc), to test agents' ability to use commonsense knowledge while interacting with the world. the hypothesis behind twc observation: you've entered a kitchen. you see a dishwasher and a fridge. here's a dining table. you see a red apple and a dirty plate on the table. state graph construction commonsense knowledge retrieval goal: clean up the kitchen 1. take the apple 2. put the apple in the fridge 3. take the dirty plate 4. put the plate in the dishwasher best action trajectory state graph commonsense graph bidirectional knowledge graph attention (bike) apple fridge plate dishwasher table kitchen on on in in in apple fridge plate dishwasher atlocation atlocation fruit isa dish isa washing dishes usedfor figure 1: an illustration of a tbg that requires both the state representation of the game as well as the external commonsense knowledge for efficient exploration and learning the best action trajectory. the observation text feeds into the state and commonsense graphs; and the best action trajectory is computed based on information from both graphs. in this paper, we posit that to efficiently act in such text-based gaming environments, an agent must be able to effectively track the state of the game, and use that to jointly retrieve and leverage the relevant commonsense knowledge. for example, commonsense knowledge such as apple should be placed in the refrigerator would help the agent to act closer to the optimal behavior; whereas state information like apple is on the table would help the agent plan more efficiently. thus, we propose a technique to: (a) track the state of the game in the form of a symbolic graph that represents the agent's current belief of the state of the world (ammanabrolu and hausknecht, 2020a; @xcite ; (b) retrieve the relevant commonsense knowledge from conceptnet @xcite , and (c) jointly leverage the state graph and the retrieved commonsense graph. this combined information is then used to select the optimal action. finally, we demonstrate the performance of our agent against state of the art baseline agents on the twc environment. ## related work text-based reinforcement learning text-based games have recently emerged as a promising framework to drive advances in rl research. prior work has explored text-based rl to learn strategies based on an external text corpus @xcite or from textual observations @xcite . in both cases, the text is analyzed and control strategies are learned jointly using feedback from the gaming environment. zahavy et al. (2018 ) proposed the action-elimination deep q-network (ae-dqn), which learns to classify invalid actions to reduce the action space. the use of the commonsense and state graph in our work has the same goal of down-weighting implausible actions by jointly reasoning over the state of the game and prior knowledge. recently, côté et al. (2018) introduced textworld and murugesan et al. (2021) proposed textworld commonsense (twc), a textbased gaming environment which requires agents to leverage prior knowledge in order to solve the games. in this work, we build on the agents of murugesan et al. (2021) and show that prior knowledge and state information are complementary and should be learned jointly. kg-based state representations a recent line of work in tbgs aims at enhancing generalization performance by using symbolic representations of the agent's belief. notably, ammanabrolu and riedl (2019) proposed kg-dqn and ammanabrolu and hausknecht (2020b) proposed kg-a2c. the idea behind both approaches is to represent the game state as a belief graph. recently, adhikari et al. ( 2020 ) proposed the graph-aided transformer agent (gata), an approach to construct and update a latent belief graph during planning. our work integrates these graph-based state representations with a prior commonsense graph that allows the agent to better model the state of the game using prior knowledge. ## model & architecture tbgs can be framed as partially observable markov decision processes (pomdps) (spaan, 2012) denoted s, a, o, t, e, r , where: s denotes the set of states, a denotes the action space, o denotes the observation space, t denotes the state transition probabilities, e denotes the conditional observation emission probabilities, and r : s × @xmath0. the observation o t at time step t depends on the current state. both observations and actions are rendered in text. the agent receives a reward at every time step t: r @xmath1, a t ), and the agent's goal is to maximize the expected discounted sum of rewards: e[∑ t γ t r t ], where γ ∈ [0, 1] is a discount factor. the high-level architecture of our model contains three major components: (a) the input encoder; (b) a graph-based knowledge extractor; and (c) the action prediction module. the input encoding layers are used to encode the observation o t at time step t and the list of admissible actions using grus @xcite . the graph-based knowledge extractor collects relevant knowledge from complementary knowledge sources: the game state, and external commonsense knowledge. we allow information from each knowledge source to guide and direct better representation learning for the other. recent efforts have demonstrated the use of primarily two different types of knowledge sources for textworld rl agents. a state graph (sg) captures state information @xcite about the environment represented via a language-based semantic graph. the example in figure 2 shows that information such as apple ## knowledge integration using bike the aforementioned graph-based knowledge extractor produces m entities (c 1 t , c 2 t , • • • , c m t ) for the commonsense graph (cg); and n entities (s 1 t , s 2 t , • • • , s n t ) for the state graph (sg). note that the entities extracted for the cg are based on the vocabulary used in conceptnet, and may not necessarily have the same set of entities as the sg (figure 1 ). we embed the extracted entities in both graphs using numberbatch @xcite . we then encode these graph representations using a graph attention network (gat) @xcite . gat allows the node entities s t and c t within the graphs g s t and g c t respectively to share information among each other by message passing. we then integrate sub-graphs extracted from the previous steps to improve the agent's exploration strategy. inspired from bidirectional attention mechanism in qa @xcite , we use bike attention mechanism between g s t and g c t to fuse the knowledge from these two graphs. the information flow across the graphs allows the model to learn commonsense-aware state graph representations, and state-aware commonsense knowledge graph representations. to implement this, we compute a graph similarity matrix @xmath2. s i @xmath3, c j t ) captures how each node s i t in the graph g s t is linked to a node c j t in the other graph g c t , and vice versa. here f is a learnable function that maps s i t and c j t to a similarity score. this allows us to measure the similarity between (for instance) apple observed in the state graph and apple observed in the commonsense graph. we compute the state-to-commonsense graph attention values a by taking a softmax along the rows of s: this signifies the attention bestowed by each state graph node on the nodes of the commonsense graph. similarly, we compute the commonsense-tostate graph attention values ā by taking a softmax along the columns of s. we capture the relevant knowledge in the commonsense graph g t c by updating the state representations si t . we compute the updated state representation as: and g is a learnable function that maps the concatenated s i t , si t , and si t to an updated state representation. finally, we use the general attention between the o t and the state graph entities s t+1 to get the state graph representation g s t+1 @xcite . we perform a similar process for the commonsense-tostate graph attention and obtain the commonsense graph representation: g c t+1 . we select the relevant action by computing an attention over the actions: h(o t , a i t , g s t+1 , g c t+1 ); where h is a learnable function that projects the concatenation o t , a i t , g s t+1 , g c t+1 to the attention score for the i th action. ## experiments we generate a set of games with 3 difficulty levels using the twc (murugesan et al., 2021) framework: (i) easy level, which has 1 room containing 1 to 3 objects; (ii) medium level, which has 1 or 2 rooms with 4 or 5 objects; and (iii) hard level, a mix of games with a high number of objects (6 or 7 objects in 1 or 2 rooms) or high number of rooms (3 or 4 rooms containing 4 or 5 objects). we compare 5 text-based rl agents: (a) a textonly agent (text), which selects the best action based only on the encoding of the history of observations; (b) drrn @xcite , which relies on the relevance between the observation and action spaces; (c) an agent enhanced with access to an external commonsense knowledge graph (+commonsense) @xcite ; (d) an agent that, following ammanabrolu and hausknecht (2020a), models the state of the world as a symbolic graph (+state); and (e) the agent (bike) described in section 3, which relies on both state and commonsense graph representations. the agents are trained over 100 episodes with a 50-step maximum. all policies are learned using actor-critic @xcite . ## conclusion we hypothesize that in order to be sample-efficient in text-based games, agents must be able to jointly track the state of the game and retrieve the relevant commonsense knowledge. we proposed a technique that models both forms of knowledge as graphs and combines them using a novel graph coattention mechanism. we show that the resulting agent is more sample-efficient than approaches that consider neither or only one of these graphs. compared our agents on two test sets: (in) uses the same entities as the training set, and (out) uses entities that were not included in the training set. the experimental results show that the bike agent generalizes better than all the baselines across the 3 difficulty levels.