Interactive Machine Comprehension with Dynamic Knowledge Graphs (2021)

Abstract: Interactive machine reading comprehension (iMRC) is machine comprehension tasks where knowledge sources are partially observable. An agent must interact with an environment sequentially to gather necessary knowledge in order to answer a question. We hypothesize that graph representations are good inductive biases, which can serve as an agent’s memory mechanism in iMRC tasks. We explore four different categories of graphs that can capture text information at various levels. We describe methods that dynamically build and update these graphs during information gathering, as well as neural models to encode graph representations in RL agents. Extensive experiments on iSQuAD suggest that graph representations can result in significant performance improvements for RL agents.

## introduction machine reading comprehension (mrc) has gathered wide interest from the nlp community in recent years. it serves as a way to benchmark a system's ability to understand and reason over natural language. typically, given a knowledge source such as a document, a model is required to read through the knowledge source to answer a question about some information contained therein. in the extractive qa paradigm, in particular, answers are typically sub-strings of the knowledge source @xcite @xcite . models are thus required to select a span from the knowledge source as their prediction. a recent line of work known as interactive machine reading comprehension (imrc) features interactive language learning and knowledge acquisition @xcite . it shifts the focus of mrc research towards a more realistic setting where the knowledge sources (environments) are partially observable. under this setting, agents must iteratively interact with the environment to discover necessary information in order to answer the questions. the sequence of interactions between an agent and the environment may resemble the agent's reasoning path, rendering a higher level of interpretability in the agent's behaviour. the trajectories of interactions can also be seen as procedural knowledge, which potentially brings agents extra generalizability (humans do not necessarily know the answer to a question immediately, but they know the procedure to search it). compared to many static mrc datasets, where the entire knowledge source (e.g., a paragraph in squad) is presented to the model immediately, the imrc setting may alleviate the risk of learning shallow pattern matching @xcite . on a parallel track, there have been a plethora of studies that leverage graphs in mrc. multiple linguistic features have been explored to help construct graphs, such as coreference @xcite , entities @xcite @xcite and semantic roles @xcite . in related areas such as vision-and text-based games, prior works also attempt to build implicit and explicit graphs to encode data from various types of modalities @xcite @xcite . all of these works, covering domains from static mrc to sequential decision making, suggest that graph representations can facilitate model learning. this gives us a strong motivation to leverage graph representations in the imrc setting. we hypothesize that graph representations are good inductive biases, since they can serve naturally as a memory mechanism to help rl agents tackle partial observability in imrc tasks. we develop an agent that can dynamically update new information into a graph representation at every step, the agent integrates information from both text and graph modalities to make decisions. the main contributions of this work are as follows: ## problem setting we follow the imrc setting @xcite , where given an environment consisting of a partially observable document and a question, an agent needs to sequentially interact with the environment to discover necessary information and then answer the question. the imrc paradigm reformulates existing mrc datasets (e.g., squad) into interactive environments by occluding most parts of their documents. a set of commands are defined to help agents reveal glimpses of the hidden documents. imrc can be seen as a controllable simulation to a family of complex real world environments, where knowledge sources are partially observable yet easily accessible by design through interactions. one such example is the internet, where humans can efficiently navigate through keywords and links to retrieve only the necessary information, rather than reading through the entire collection of websites. while imrc shares some common properties with multi-step retrieval @xcite and open-domain qa @xcite , the focus here is to push the boundaries of information-seeking agents @xcite ) from an rl/navigation perspective. formally, an imrc data-point (game) is a discrete-time, partially observable markov decision process (pomdp) defined by (s, t, a, ω, o, r, γ). at game step t, the environment state s @xmath0, as well as which subset of the sentences has been revealed to the agent. the agent perceives text information as its observation, o @xmath1, which depends on the environment state with probability o(o t |s t ). the agent issues an action a @xmath2, resulting in a state transition s t+1 with probability t (s t+1 |s t , a t ) in the environment (i.e., a new sentence is shown to the agent). based on its actions, the agent receives rewards r @xmath3, a t ). the agent's objective is to maximize the expected discounted sum of rewards @xmath4, where γ ∈ [0, 1] is the discount factor. ## methodology in this work, we adopt the agent proposed in @xcite as baseline, as shown in figure 1 . we propose to add a novel graph updater module and a graph encoding layer into the pipeline. specifically, at game step t, the graph updater takes the text observation o t and the graph g t-1 from previous step as input and generates a new graph g t . subsequently, the graph is encoded into hidden states, which is later aggregated with text representations. note that distinct from fully observed knowledge graphs (kgs) in static mrc works, our graphs are dynamically generated, i.e., at every interaction step, our agent can update information from the new observation into its graph representations. in this section, we will first introduce the key methods we use to generate and update the graph representations. later on, we will describe a graph neural network (gnn)-based graph encoder which encodes the information carried by the graphs. for the common components shared with imrc, we refer readers to @xcite or appendix a for detailed information. notations we denote a graph generated at game step t as g @xmath5, e t ), where v t and e t represent the set of vertices (nodes) and edges (relations). all graphs are directed by default. for two nodes @xmath6, we denote the connection from i to j as e @xmath7. we represent graph g as an adjacency tensor, with the size of r × n × n , where r and n denote the number of relations and nodes, respectively. this tensor can either be binary or real-valued depending on graph type. ## experiments and results we conduct experiments on the isquad dataset @xcite to answer three key questions: experiment setup: the isquad dataset @xcite is an interactive version of the squad dataset @xcite , which consists of 82k/5k/10k environments for training, validation, and testing. as described in section 2, isquad contains two difficulty levels and three finer-grained query type settings, all of which influence an rl agent's action space. the environment provides an observation queue with k memory slots depending on different configurations, where @xmath8, 3, 5}. the observation queue stores the k most recent observation sentences to alleviate difficulties caused by partial observability. note in configuration where @xmath9, there is no history information stored. inherited from the original squad dataset, an agent is evaluated by the f 1 score between its predicted answer and the ground-truth answers. we compare our agents equipped with graph representations against imrc scores reported in @xcite , specially, we also report an agent m's relative improvement over imrc: for all experiment settings, we train the agent with three different random seeds. we compute an agent's test score using the model checkpoint that achieves the best validation score. a1: graph representations indeed help, and ensemble is an useful strategy. intuitively, a dynamically maintained graph can serve as an agent's episodic memory. therefore, the less information is provided by the environment, the more useful the graphs can be. we first investigate our graph aided agent's performance on the game configuration where only single memory slot is available. this is arguably the most difficult configuration in imrc, where any valid action can lead to a completely different observation (a new sentence). as a result, agents needs to rely on its own memory mechanism. as shown in following standard strategy of model ensemble in mrc works, we test the ensemble of the four graphs. specifically, taking four individual agents, each trained with its corresponding graph types, we mix their decisions during test. during the information gathering phase, we sum up the four agents' output probabilities, including the probabilities over action words (i.e., previous, next, ctrl+f, stop) and the probabilities over the query tokens. the four agents consequently take the action with the max summed probabilities to keep interacting with the environment. during the question answering phase, we also sum up the output probabilities (over tokens in the sentence where the agents stop), and generate answers accordingly. surprisingly, we find that the ensemble greatly boosts agent's performance. as shown in a2: graph representations remain helpful even with explicit memories. as mentioned above, in some configurations, the isquad environment provides an observation queue that caches most recent few observations as an explicit memory mechanism. a natural thing to explore is that if the advantages of equipping graph representations tend to diminish when the partial observability of the environments decreases. we train and test our agent using isquad's configurations where 3 or 5 memory slots are available (i.e., at game step t, the input o t to the agent is the concatenation of the most recent 3 or 5 sentences it has seen). from given the observation that graph representations seem still helpful even with explicit memories, we further compare graph as memory mechanism against the explicit memory slots provided by isquad environments. comparing our best graph aided agent (receiving single sentence as input) against imrc (receiving 3 and 5 sentences as input), we find our agent achieves a %13.03 and %13.47 of relative improvements over imrc. this suggests that the design of the memory mechanism plays a big role in the interactive reading comprehension tasks. although the concatenation of memory slots may provide as much amount of information, the inductive bias of graph representations are stronger. ## related work mrc has become an ever-growing area in the past decade, especially since the success of deep neural models. like an adversarial game, researchers release new datasets @xcite @xcite @xcite @xcite @xcite and novel models @xcite @xcite @xcite one after another. since the flourishing of large scale pretrained language models such as bert @xcite , roberta @xcite and xl-net @xcite , performance of neural models on mrc datasets have improved greatly. while some researchers believe models have achieved human-level performance, others argue that there have been biases or trivial cues injected into mrc datasets unconsciously @xcite @xcite @xcite . these biases may cause models to learn shallow pattern matching, rather than deep understanding and reasoning skills. graph construction is also a thriving direction lies at the intersection of multiple areas such as information extraction @xcite , knowledge base @xcite , logical reasoning @xcite and representation learning @xcite . leveraging automatically constructed graph representations in static mrc has been shown effective, researchers use a wide range of linguistic features to help constructing graphs. dhingra et al. (2018 2020 ) leverage mentions of entities and zheng and kordjamshidi (2020) build srl graphs using parsers. in the context of rl, prior work have also shown that constructing graph representations from other modalities can be helpful to solve tasks in interactive environments @xcite @xcite . ## broader impact our work is a proof-of-concept study, we use a relatively simple and restricted (in terms of both observations and actions) qa dataset, isquad, for both training and evaluation. although the current version of our work might have limited consequences for society, we believe that taking a broader view of our work can be beneficial by preventing our future research from causing potential social and ethical concerns. similar to many rl-based systems, the information gathering module of our agent is optimized solely on its performance w.r.t. the final metric, without much constraints on its behavior at each game step. this can potentially make the system vulnerable since the rl agent may develop undesirable strategies that optimize the final metric. in our current setting, the action space of the information gathering module is restricted (see section 2). however, if we consider a more general setting, e.g., to equip the agent with a larger action space by allowing it to generate a sequence of tokens as the query to the ctrl+f action, we have to be extra careful about the aforementioned side effects caused by rl training. for instance, the agent may develop unfavorable behaviors such as forgetting proper syntax, abusing certain pronouns, to optimize its final rewards. ## conclusion we explore to leverage graph representations in the challenging imrc tasks. we investigate different categories of graph structures that can capture text information at various levels. we describe methods that dynamically generate the graphs during information gathering. experiment results show that graph representations provide consistent improvement across settings. this evinces our hypothesis that graph representations are proper inductive biases in imrc.