Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions? (2021)

Abstract: Multi-hop question answering (QA) requires a model to retrieve and integrate information from multiple passages to answer a question. Rapid progress has been made on multi-hop QA systems with regard to standard evaluation metrics, including EM and F1. However, by simply evaluating the correctness of the answers, it is unclear to what extent these systems have learned the ability to perform multi-hop reasoning. In this paper, we propose an additional sub-question evaluation for the multi-hop QA dataset HotpotQA, in order to shed some light on explaining the reasoning process of QA systems in answering complex questions. We adopt a neural decomposition model to generate sub-questions for a multi-hop question, followed by extracting the corresponding sub-answers. Contrary to our expectation, multiple state-of-the-art multi-hop QA models fail to answer a large portion of sub-questions, although the corresponding multi-hop questions are correctly answered. Our work takes a step forward towards building a more explainable multi-hop QA system.

## introduction rapid progress has been made in the field of question answering (qa), thanks to the release of many large-scale, high-quality qa datasets. early datasets @xcite @xcite @xcite mainly consist of single-hop questions, where an answer with supporting justification can be found within a short segment of text. these benchmarks focus on evaluating qa models' ability to perform local pattern matching between a passage and a question. existing models @xcite have achieved super-human performance. recently, multi-hop qa datasets @xcite @xcite have gained increasing attention. they require models to retrieve multiple pieces of supporting evidence from different documents and to reason over the evidence collected to answer a question. the standard evaluation metrics of qa datasets include exact match (em) and f1 scores averaged over the test set. hotpotqa @xcite also provides sentence-level supporting facts required for reasoning. however, providing supporting sentences is not sufficient for us to interpret the choice of an answer for end-to-end complex qa systems. it is unclear whether the systems have performed the desired multi-hop reasoning to reach the correct answer. in this work, we propose an additional evaluation scheme to test multi-hop qa systems' performance on answering the single-hop sub-questions of a multi-hop question. when designing a multihop question, we expect it to require qa models to retrieve a chain of sentences as evidence and then reasoning over them to answer the question. evaluating qa models on sub-questions helps us to understand their behavior on each hop of the reasoning process. in addition, it evaluates whether multi-hop qa models can generalize well on simpler questions. figure 1 presents an illustrating example. a successful complex qa model should be able to answer the two sub-questions "which movie stars arnold schwarzenegger as a former new york police detective" and "what year did guns n roses perform a promo for end of days" if it understands the underlying reasoning process for the original multi-hop question. we focus on the hotpotqa @xcite ) dataset under the distractor setting, in which multihop questions are asked over several wikipedia paragraphs. we create the evaluation dataset by generating the sub-questions and then extracting their answers automatically. the candidate subquestions and intermediate answers are then manually verified, which results in 1,000 sub-question evaluation examples. it is surprising to find that all three top-performing models which we experiment with fail to answer a large portion of sub-questions (49.8% to 60.4%), although their corresponding multi-hop questions are correctly answered. previous work has investigated the necessity of multi-hop reasoning on hotpotqa dataset. @xcite construct distracting paragraphs adversarially to demonstrate that models learn to exploit reasoning shortcuts to locate the answer rather than performing multi-hop reasoning. @xcite show that a sentence-factored model can solve a large number of questions in hot-potqa, suggesting multi-hop reasoning is not really needed. min et al. (2019a) also achieve similar result using a single-hop bert-based model. our sub-question evaluation is complementary to these approaches. while existing work shows the lack of multi-hop reasoning by limiting or adding text input to qa models, we provide sub-questions and intermediate answers explicitly to interpret model behavior on each hop of the reasoning process. it can be used as a complementary metric to ensure that models which can correctly answer both intermediate sub-questions and the final multi-hop question actually go through the reasoning steps as desired. our work takes a step forward towards building a more explainable multi-hop qa system. ## construction of evaluation examples in this section, we introduce our semi-automatic approach to generate two sub-questions and their corresponding answers for multi-hop questions from the hotpotqa dataset. as shown in figure 1 , the evaluation examples are generated in three steps. first, we decompose each source question into several sub-strings by predicting the breaking points and post-process them to generate two sub-questions. then, the answers for the subquestions are extracted from the paragraphs using some heuristics. lastly, the candidate evaluation examples generated are sent for human verification. we first introduce the hotpotqa dataset and then elaborate on each step of the construction pipeline. ## experiments and results in order to interpret the behavior of existing models on each hop of the reasoning process required for multi-hop questions and to determine their ability to answer simple questions, we perform sub-question evaluation on three published topperforming qa models with publicly available open-source code: dfgn @xcite , de-comprc @xcite , and cogqa @xcite . for all experiments, we measure em and f1 scores for q, sub q 1 , and sub q 2 on 1,000 human-verified examples. to measure the correctness of a predicted answer, we first use exact string match as the only metric. however, during error analysis, we find that many predicted answers that partially match the gold answers should also be regarded as correct. some representative examples are shown in ## conclusion we propose a new way to interpret whether multihop qa systems explore the multiple steps of reasoning over the evidence as desired by asking subquestions. an automatic approach is designed to generate sub-questions for a multi-hop question. on a human-verified test set, our experiments demonstrate that top-performing multi-hop qa models fail to answer a large portion of subquestions whose parent multi-hop questions can be correctly answered. we believe that progress on building complex qa systems that truly understand multi-hop reasoning is only possible if the evaluation metrics reward this kind of behavior. as an initial step towards a more explainable qa system, we hope our work would motivate the construction of multi-hop qa datasets with explicit reasoning paths annotated and the development of better multi-hop qa models.