D iff G - RL : Leveraging Difference between Environment State and Common Sense (2022)

Abstract: Taking into account background knowledge as the context has always been an important part of solving tasks that involve natural language. One representative example of such tasks is text-based games, where players need to make decisions based on both description text previously shown in the game, and their own background knowledge about the language and common sense. In this work, we investigate not simply giving common sense, as can be seen in prior research, but also its effective usage. We assume that a part of the environment states different from common sense should constitute one of the grounds for action selection. We propose a novel agent, DiffG-RL, which constructs a Difference Graph that organizes the environment states and common sense by means of interactive objects with a dedicated graph encoder. DiffG-RL also contains a framework for extracting the appropriate amount and representation of common sense from the source to support the construction of the graph. We validate DiffG-RL in experiments with text-based games that require common sense and show that it outperforms baselines by 17% of scores. We will make our code publicly available.

## introduction taking into account background knowledge as the context has always been an important yet challenging part of solving tasks that involve natural language. one illustrative example of such challenges is text-based games. text-based games are computer games where game states and action spaces are represented in pure texts. to play them, players have to not only understand in-game texts correctly but also make appropriate action decisions from given options according to the context. computational agents required to solve such games naturally arise in the form of natural language processing (nlp) systems trained with reinforcement learning figure 1 : an example of text-based games. our proposed diffg-rl summarizes the difference between the environment states and common sense in a graph and uses it as a basis for predicting a plan of action. previous approaches @xcite @xcite @xcite have used external knowledge to constraint agents' action outputs in order to shrink the size of search space. recently, @xcite and @xcite utilized human common sense which improved sample efficiency and enabled agents to perform look-ahead planning. however, these approaches have not yet clarified how common sense should actually be used. specifically, a huge amount of common sense is given at once, regardless of the environment states, and the correspondence between the states and common sense is unclear. this prevents agents from learning which common sense to use in which state, and the agents remember only the results after using common sense. in this paper, as usage of common sense, we assume that differences between environment states and common sense can provide a basis for action selection and further improve sample efficiency. for example, consider the environment state of "dirty fork on the floor" and the common sense of "dirty fork should be in the dishwasher". the difference between floor and dishwasher in the location of the dirty fork helps agents plan their actions to pick it up from the floor and put it in the dishwasher. we construct a difference graph that maps environment states to common sense and explicitly represents their differences. further, we develop an encoder dedicated to this graph and propose an agent that can effectively concentrate on learning which common sense to use in which state. an example is shown in fig. 1 . inevitable problems in constructing the difference graph are extracting the right amount and unifying the representation. for the first problem, large amounts of common sense cannot be encoded, and small amounts are insufficient for learning. in prior work, the amount of common sense is reduced by extracting only common sense that contains representations that exactly match the objects that appear in the environment, but in some tasks, there may be no common sense available due to mismatches of representations. in contrast, we extract the appropriate amounts of common sense based on semantics and the circumstances, independent of linguistic representations. for the second problem, to help agents recognize the difference between common sense and environmental states, their representations should be aligned. therefore, we propose an extraction framework for common sense that includes an acquisition of appropriate amounts based on meanings and circumstances and a representation transformation that facilitates the mapping to environment states. our contributions in this work are as follows. common sense extractor environment states extractor difference graph node encoder difference graph encoder next action a t+1 admissible commands a t take a bell pepper from the table. take a dirty fork from the table. observation o t you've entered a kitchen. you see a dishwasher and a fridge. command encoder scorer weight share difference encoder action selector dirty fork dishwasher bell pepper fridge common sense sourcec we build on this to extract the states of interactive objects from observation text by using the abstract meaning representation (amr) parser @xcite . nodes connected to the interactive objects by positional relationships ("on," "in," etc.) from amr are extracted as the current states. we also add a node representing the player "you" and use it when the interactive objects are in the inventory. at each step, the entities e t , the interactive objects i t , and their current states are updated on the basis of the observations, and the states of all interactive objects are tracked. in this section, we first present an overview of our proposed agent, called diffg-rl, and then propose a framework for the extraction of common sense to facilitate the comparison of differences between environment states and common sense. finally, we introduce the difference graph and its encoder, which provides a representation of the difference. ## conclusion in this work, we investigated the difference between the environment state and common sense as a basis for rl agent decision-making in textbased games. we proposed diffg-rl, a novel agent that constructs a graph that represents the difference, along with a dedicated encoder, also contains a common sense extraction framework to obtain the appropriate amount and representation of common sense to facilitate the comparison between the environment states and the common sense. our experimental results showed that diffg-rl outperformed baselines that used only a knowledge graph, only common sense, or a naive combination of the two. these findings demonstrate the effectiveness of the difference graph, which is a representation of the difference between the environment states and common sense, for text-based games.