Interactive Machine Comprehension with Information Seeking Agents (2020)

Abstract: Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA). We argue that this stems from the nature of MRC datasets: most of these are static environments wherein the supporting documents and all necessary information are fully observed. In this paper, we propose a simple method that reframes existing MRC datasets as interactive, partially observable environments. Specifically, we “occlude” the majority of a document’s text and add context-sensitive commands that reveal “glimpses” of the hidden text to a model. We repurpose SQuAD and NewsQA as an initial case study, and then show how the interactive corpora can be used to train a model that seeks relevant information through sequential decision making. We believe that this setting can contribute in scaling models to web-level QA scenarios.

## introduction many machine reading comprehension (mrc) datasets have been released in recent years @xcite @xcite @xcite to benchmark a system's ability to understand and reason over natural language. typically, these datasets require an mrc model to read through a document to answer a question about information contained therein. the supporting document is, more often than not, static and fully observable. this raises concerns, since models may find answers simply through shallow pattern matching; e.g., syntactic similarity between the words in questions and documents. as question: what was the harvard endowment total in 2011 ? harvard has the largest university endowment in the world . ## related works skip-reading @xcite @xcite is an existing setting in which mrc models read partial documents. concretely, these methods assume that not all tokens in the input sequence are equally useful, and therefore learn to skip irrelevant tokens. since skipping decisions are discrete, the models are often optimized by the reinforce algorithm @xcite . for example, the structural-jump-lstm @xcite learns to skip and jump over chunks of text, whereas @xcite designed a qa task where the model reads streaming data without knowing when the question will be provided. skipreading approaches are limited in that they only consider jumping forward over a few consecutive tokens. based on the assumption that a single pass of reading may not provide sufficient information, multi-pass reading methods have also been studied @xcite . compared to skip-reading and multi-pass reading, our work enables an agent to jump through a document in a more dynamic manner, in some sense combining aspects of skip-reading and rereading. specifically, an agent can choose to read forward, backward, or to jump to an arbitrary position depending on the query. this also distinguishes the model we develop in this work from reasonet @xcite , a model that decides when to stop forward reading. recently, there has been various work on and around interactive environments. for instance, @xcite proposed webnav, a tool that automatically transforms a website into a goaldriven web navigation task. they train a neural agent to follow traces using supervised learning. @xcite proposed golden retriever, an iterative retrieve-and-read system that answers complex open-domain questions, which is also trained with supervised learning. although an effective training strategy, supervised learning requires either human labeled or heuristically generated trajectories. however, there often exist multiple trajectories to solve each question, many of which may not be observed in the supervised data since it is difficult to exhaust all valid trajectories. generalization can be limited when an agent is trained on such data. @xcite introduced a collection of synthetic tasks to train and test informationseeking capabilities in neural models. @xcite proposed an information extraction system that acquires and incorporates external evidence to improve extraction accuracy in domains with limited data. @xcite proposed a dqn-based agent that leverages the (tree) structure of documents and navigates across sentences and paragraphs. imrc is distinct from this body of literature in that it does not depend on extra meta information to build tree structures, it is partially-observable, and its action space is as large as 200,000 (much larger than, e.g., the 5 query templates in @xcite and tree search in @xcite ). our work is also inspired directly by qait @xcite , a set of interactive question answering tasks developed on text-based games. however, qait is based on information gathering +1 encoder action generator question answerer imrc ≠ stop ＝ stop answer question question answering figure 1: a demonstration of the proposed imrc pipeline, in which the agent is illustrated as a shaded area. at a game step t, it encodes the question and text observation into hidden representations m t . an action generator takes m t as input to generate commands to interact with the environment. if the agent generates stop at this game step, m t is used to answer question by a question answerer. otherwise, the imrc environment will provide new text observation in response to the generated action. broadly speaking, our work is also linked to the query reformulation (qr) task in information retrieval literature @xcite . specifically, qr aims to automatically rewrite a query so that it becomes more likely to retrieve relevant documents. our task shares the spirit of iterative interaction between an agent (reformulator in qr) and an environment. however, the rewritten queries in qr tasks keep the semantic meaning of the original queries, whereas in our task, actions and queries across different game steps can change drastically -since our task requires an agent to learn a reasoning path (trajectory) towards answering a question, rather than to search the same concept repeatedly. ## imrc: making mrc interactive the isquad and inewsqa datasets are based on squad v1.1 foot_0 @xcite and newsqa @xcite . both original datasets share similar properties. specifically, each data-point consists of a tuple, {p, q, a}, where p represents a paragraph, q a question, and a is the answer. the answer is a word span defined by head and tail positions in p. newsqa is more chal-lenging because it has a larger vocabulary, more difficult questions, and longer source documents. every paragraph p is split into a list of sentences @xmath0, s 2 , ..., s n }, where n stands for number of sentences in p. at the start of a question answering episode, an agent observes the question q, but rather than observing the entire paragraph p, it sees only the first sentence s 1 while the rest is withheld. the agent must issue commands to reveal the hidden sentences progressively and thereby gather the information needed to answer q. the agent should decide when to stop interacting and output an answer, but the number of interaction steps is limited. 3 once the agent has exhausted its step budget, it is forced to answer the question. ## baseline agent as a baseline agent, we adopt qa-dqn @xcite , we modify it to enable extractive query generation and question answering. as illustrated in figure 1 , the baseline agent consists of three components: an encoder, an action generator, and a question answerer. more precisely, at a step t during the information gathering phase, the encoder reads observation string o t and question string q to generate the attention aggregated hidden representations m t . using m t , the action generator outputs commands (depending on the mode, as defined in § 3.2) to interact with imrc. the information-gathering phase terminates whenever the generated command is stop or the agent has used up its move budget. the question answerer takes the hidden representation at the terminating step to generate head and tail pointers as its answer prediction. ## experimental results in this study, we focus on four main aspects: regarding the four aspects, we report the baseline agent's training performance followed by its generalization performance on test data. we use dqn and a2c to refer to our baseline agent trained with dqn and a2c, respectively. we set the maximum number of episodes (data points) to be 1 million, this is approximately 10 epochs in supervised learning tasks given the size of datasets. the agent may further improve after 1 million episodes, however we believe some meaningful and interesting trends can already be observed from the results. besides, we hope to keep the wall clock time of the task reasonable foot_2 to encourage the community to work on this direction. ## discussion and future work in this work, we propose and explore the direction of converting mrc datasets into interactive, partially observable environments. we believe information-seeking behavior is desirable for neural mrc systems when knowledge sources are partially observable and/or too large to encode in their entirety, where knowledge is by design easily accessible to humans through interaction. our idea for reformulating existing mrc datasets as partially observable and interactive environments is straightforward and general. it is complementary to existing mrc dataset and models, meaning almost all mrc datasets can be used to study interactive, information-seeking behavior through similar modifications. we hypothesize that such behavior can, in turn, help in solving real-world mrc problems involving search. as a concrete example, in real world environments such as the internet, different pieces of knowledge are interconnected by hyperlinks. we could equip the agent with an action to "click" a hyperlink, which returns another webpage as new observations, thus allowing it to navigate through a large number of web information to answer difficult questions. imrc is difficult and cannot yet be solved, however it clearly matches a human's informationseeking behavior compared to most static and fullyobservable laboratory mrc benchmarks. it lies at the intersection of nlp and rl, which is arguably less studied in existing literature. for our baseline, we adopted off-the-shelf, top-performing mrc and rl methods, and applied a memory mechanism which serves as an inductive bias. despite being necessary, our preliminary experiments do not seem sufficient. we encourage work on this task to determine what inductive biases, architectural components, or pretraining recipes are necessary or sufficient for mrc based on information-seeking. our proposed setup presently uses only a single word as query in the ctrl+f command in an abstractive manner. however, a host of other options could be considered in future work. for example, a multi-word query with fuzzy matching is more realistic. it would also be interesting for an agent to generate a vector representation of the query in some latent space and modify it during the dynamic reasoning process. this could further be used to retrieve different contents by comparing with pre-computed document representations (e.g., in an open-domain qa dataset), with such behavior tantamount to learning to do ir. this extends traditional query reformulation for open-domain qa by allowing to drastically change the queries without strictly keeping the semantic meaning of the original queries.