Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning (2020)

Abstract: Has there been real progress in multi-hop question-answering? Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting facts. This limits our ability to measure true progress and defeats the purpose of building multi-hop QA datasets. We make three contributions towards addressing this. First, we formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of contrastive support sufficiency , we introduce an automatic transformation of existing datasets that reduces the amount of disconnected reasoning. Third, our experiments suggest that there hasn’t been much progress in multi-hop QA in the reading comprehension setting. For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation substantially reduces disconnected reasoning (19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction.

## introduction multi-hop question answering requires connecting and synthesizing information from multiple facts in the input text, a process we refer to as multifact reasoning. prior work has, however, shown that bad reasoning models, ones that by design do not connect information from multiple facts, can achieve high scores because they can exploit specific types of biases and artifacts (e.g., answer type * early portion of this work was done during the first author's internship at allen institute for ai. ## related work multi-hop reasoning: many multifact reasoning approaches have been proposed for hotpotqa and similar datasets @xcite . these use iterative fact selection @xcite @xcite , graph neural networks @xcite @xcite , or simply cross-document self-attention @xcite to capture inter-paragraph interaction. while these approaches have pushed the state of the art, the extent of actual progress on multifact reasoning remains unclear. identifying dataset artifacts: several works have identified dataset artifacts for tasks such as nli @xcite , reading comprehension @xcite , and even multi-hop reasoning @xcite . these artifacts allow models to solve the dataset without actually solving the underlying task. on hotpotqa, prior work has shown existence of models that identify the support @xcite and answer @xcite by operating on each paragraph or sentence independently. we, on the other hand, estimate the amount of disconnected reasoning in any model and quantify the cheatability of answer and support identification. ## measuring disconnected reasoning this section formalizes the dire condition, which captures what it means for a model to employ disconnected reasoning, and describes how to use this condition to probe the amount of disconnected reasoning performed by a given model, and the extent of such reasoning possible on a dataset. a good multifact reasoning is one where information from all the supporting facts is meaningfully synthesized to arrive at an answer. the precise definition for what constitutes meaningful synthesis is somewhat subjective; it depends on the semantics of the facts and the specific question at hand, making it challenging to devise a measurable test for the amount of multifact (or non-multifact) reasoning done by a model or needed by a dataset. previous works have used the answer prediction task (i.e., identifying the correct answer) and the supporting fact identification task (identifying all facts supporting the answer) as approximate tests of multifact reasoning. we argue that, at a minimum, good multifact reasoning requires connected reasoning-one where information from at least one supporting fact is connected to the selection and use of information from other supporting facts. consider the example question in figure 1 . a good multifact reasoning will look for a supporting fact that mentions when the cold war started ( ) and use information from this fact (year 1947) to select the other supporting fact mentioning the country that got independence ( ) (or vice versa). a bad multifact reasoning model, however, can cheat on answer prediction by only looking for a fact that mentions a country getting independence at some time (mentioned in ), without connecting this to when the cold war started (mentioned in ). similarly, the model can also cheat on supporting fact identification by treating it as two independent sub-tasks-one returning a fact mentioning the time when a country got independence, and another for a fact mentioning the time when cold war started. the result of at least one of the two sub-tasks should influence the result of the other sub-task, but here it does not. this results in disconnected reasoning, where both supporting facts are identified without reference to the other. even though the precise definition of a meaningful synthesis of information is unclear, it is clear that models performing this type of disconnected reasoning cannot be considered as doing valid multifact reasoning. neither answer prediction nor support identification directly checks for such disconnected reasoning. ## reducing disconnected reasoning this section introduces an automatic transformation of a dataset to make it less cheatable by disconnected reasoning. it also defines a probe dataset for assessing how cheatable the transformed dataset is. a disconnected reasoning model does not connect information across supporting facts. this has an important consequence: when a supporting fact is dropped from the context, the model's behavior on other supporting facts remains unchanged. figure 2 illustrates this for the example in figure 1 . 5 our experiments also include inoculation (i.e., finetuning on a small fraction of the dataset) before evaluation. 6 for answer prediction with exact-match, this corresponds to m getting 1 point for correctly answering a question group in p(d) only if it correctly answers the corresponding original question in d as well. support facts identification test contrastive support sufficiency test labels labels connect? connect? connect? output output has red? & has blue? output output has red? & has blue? has red? & has blue? suppose we create an insufficient context c ′ by removing the blue fact ( ) from c (shown in the last row of figure 2 , with the removed fact crossed out). with the full context c, the cheating model discussed earlier did not use the information in the blue fact ( ) to produce the answer or to identify the red fact ( ). therefore, the absence of the blue fact in c ′ will induce no change in this model's answer or its ability to select the red fact ( ). further, to return a second supporting fact, the model would choose the next best matching fact ( ) that also indicates the start of a war and thus appears to be a reasonable choice (see bottomright of figure 2 ). without considering interaction between the two identified facts, this model would not realize that the light blue fact ( ) does not fit well with the red fact ( ) because of the year mismatch @xcite , and the two together are thus insufficient to answer q. a good multifact model, on the other hand, connects information across different supporting facts. thus, when evaluated on context c ## experiments to obtain a more realistic picture of the progress in multifact reasoning, we compare the performance of the original glove-based baseline model @xcite and a state-of-the-art transformerbased lm, xlnet @xcite on the multihop qa dataset hotpotqa @xcite . while it may appear that the newer models are more capable of multifact reasoning (based on answer and support prediction tasks), we show most of these gains are from better exploitation of disconnected reasoning. our proposed transformation reduces disconnected reasoning exploitable by these models and gives a more accurate picture of the state of multifact reasoning. to support these claims, we use our proposed dataset probes, transformations, and metrics summarized in datasets d and t(d): hotpotqa is a popular multi-hop qa dataset with about 113k questions which has spurred many models @xcite @xcite . we use the distractor setting where each question has a set of 10 input paragraphs, of which two were used to create the multifact question. apart from the answer span, each question is annotated with these two supporting paragraphs and the supporting sentences within them. as described in sec. 4.2, we use these supporting paragraph annotations as f s to create a transformed dataset t(d). models: we evaluate two models: (1) xlnet-base: since hotpotqa contexts are 10 paragraphs long, we use xlnet, a model that can handle contexts longer than 1024 tokens. we train xlnet-base to predict the answer, supporting sentences, supporting paragraphs, and the sufficiency label (only on transformed datasets). as shown in metrics: we report metrics for standard tests for hotpotqa: answer span prediction (ans), support identification (paragraph-level: supp p , sentencelevel: supp s ), as well as joint tests ans +supp p and ans +supp s . for each of these, we show f1 scores, but trends are similar for em scores. 10 these metrics correspond to m τ (q, ⋅) in section 3 and to s τ (m, d) in ## conclusions progress in multi-hop qa under the reading comprehension setting relies on understanding and quantifying the types of undesirable reasoning current models may perform. this work introduced a formalization of disconnected reasoning, a form of bad reasoning prevalent in multi-hop models. it showed that a large portion of current progress in multifact reasoning can be attributed to disconnected reasoning. using a notion of contrastive sufficiency, it showed how to automatically transform existing support-annotated multi-hop datasets to create a more difficult and less cheatable dataset that results in reduced disconnected reasoning. our probing and transformed dataset construction assumed that the context is an unordered set of facts. extending it to a sequence of facts (e.g., as in multirc @xcite ) requires accounting for the potential of new artifacts by, for instance, carefully replacing rather than dropping facts. additionally, for factual reading comprehension datasets where the correct answer can be arrived at without consulting all annotated facts in the input context, our probe will unfairly penalize a model that uses implicitly known facts, even if it correctly connects information across these facts. however, our transformation alleviates this issue: a model that connects information will have an edge in determining the sufficiency of the given context. we leave further exploration to future work. it is difficult to create large-scale multihop qa datasets that do not have unintended artifacts, and it is also difficult to design models that do not exploit such shortcuts. our results suggest that carefully devising tests that probe for desirable aspects of multifact reasoning is an effective way forward.