Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020)

Abstract: A multi-hop question answering (QA) dataset aims to test reasoning and inference skills by requiring a model to read multiple paragraphs to answer a given question. However, current datasets do not provide a complete explanation for the reasoning process from the question to the answer. Further, previous studies revealed that many examples in existing multi-hop datasets do not require multi-hop reasoning to answer a question. In this study, we present a new multi-hop QA dataset, called 2WikiMultiHopQA, which uses structured and unstructured data. In our dataset, we introduce the evidence information containing a reasoning path for multi-hop questions. The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model. We carefully design a pipeline and a set of templates when generating a question-answer pair that guarantees the multi-hop steps and the quality of the questions. We also exploit the structured format in Wikidata and use logical rules to create questions that are natural but still require multi-hop reasoning. Through experiments, we demonstrate that our dataset is challenging for multi-hop models and it ensures that multi-hop reasoning is required.

## introduction machine reading comprehension (mrc) aims at teaching machines to read and understand given text. many current models @xcite @xcite have defeated humans on the performance of squad @xcite , as shown on its leaderboard 1 . however, such performances do not indicate that these models can completely understand the text. specifically, using an adversarial method, @xcite demonstrated that the current models do not precisely understand natural language. moreover, @xcite demonstrated that many datasets contain a considerable number of easy instances that can be answered based on the first few words of the questions. multi-hop mrc datasets require a model to read and perform multi-hop reasoning over multiple paragraphs to answer a question. currently, there are four multi-hop datasets over textual data: complexwe-bquestions @xcite , qangaroo @xcite , hotpotqa @xcite , and r 4 c @xcite . the first two datasets were created by incorporating the documents (from web or wikipedia) with a knowledge base (kb). owing to their building procedures, these datasets have no information to explain the predicted answers. meanwhile, the other two datasets were created mainly based on crowdsourcing. in hotpotqa, the authors introduced the sentence-level supporting facts (sfs) information that are used to explain the predicted answers. however, as discussed in @xcite , the task of classifying sentence-level sfs is a binary classification task that is incapable of evaluating the reasoning and inference skills of the model. further, data analyses @xcite revealed that many examples in hotpotqa do not require multi-hop reasoning to solve. recently, to evaluate the internal reasoning of the reading comprehension system, @xcite proposed a new dataset r 4 c that requires systems to provide an answer and derivations. a derivation is a semi-structured natural language form that is used to explain the answers. r 4 c is created based on hotpotqa and has 4,588 questions. however, the small size of the dataset implies that the dataset cannot be used as a multi-hop dataset with a comprehensive explanation for training end-to-end systems. in this study, we create a large and high quality multi-hop dataset 2wikimultihopqa foot_0 with a comprehensive explanation by combining structured and unstructured data. to enhance the explanation and evaluation process when answering a multi-hop question on wikipedia articles, we introduce new information in each sample, namely evidence that contains comprehensive and concise information to explain the predictions. evidence information in our dataset is a set of triples, where each triple is a structured data (subject entity, property, object entity) obtained from the wikidata (see figure 1 for an example). figure 1 : example of an inference question in our dataset. the difference between our dataset and hotpotqa is the evidence information that explains the reasoning path. our dataset has four types of questions: comparison, inference, compositional, and bridge comparison. all questions in our dataset are created by using a set of predefined templates. @xcite classified the comparison questions in hotpotqa in three types: multi-hop, context-dependent multi-hop, and single-hop. based on this classification, we removed all templates in our list that make questions become single-hop or context-dependent multi-hop to ensure that our comparison questions and bridgecomparison questions are multi-hop. we carefully designed a pipeline to utilize the intersection information between the summary foot_1 of wikipedia articles and wikidata and have a special treatment for each type of question that guarantees multi-hop steps and the quality of the questions. further, by utilizing the logical rule information in the knowledge graph, such as f ather(a, b)∧f ather(b, c) ⇒ grandf ather(a, c), we can create more natural questions that still require multi-hop reasoning. we conducted two different evaluations on our dataset: difficulty and multi-hop reasoning of the dataset. to evaluate the difficulty, we used a multi-hop model to compare the performance of hotpotqa and our dataset. overall, the results from our dataset are lower than those observed in hotpotqa, while human scores are comparable on both datasets. this suggests that the number of difficult questions in our dataset is greater than that in hotpotqa. similar to @xcite , we used a single-hop bert model to test the multi-hop reasoning in our dataset. the result of our dataset is lower than the result of hotpotqa by 8.7 f1, indicating that a lot of examples in our dataset require multi-hop reasoning to be solved. through experiments, we confirmed that although our dataset is generated by hand-crafted templates and the set of predefined logical rules, it is challenging for multi-hop models and requires multi-hop reasoning. in summary, our main contributions are as follows: (1) we use wikipedia and wikidata to create a large and high quality multi-hop dataset that has comprehensive explanations from question to answer. (2) we provide new information in each sample-evidence information useful for interpreting the predictions and testing the reasoning and inference skills of the model. (3) we use logical rules to generate a simple natural question but still require the model to undertake multi-hop reasoning when answering a question. the full dataset, baseline model, and all information that we used when constructing the dataset are available at https://github.com/alab-nii/2wikimultihop . ## data analysis question and answer lengths we quantitatively analyze the properties of questions and answers for each type of question in our dataset. the statistics of the dataset are presented in ## conclusion in this study, we presented 2wikimultihopqa-a large and high quality multi-hop dataset that provides comprehensive explanations for predictions. we utilized logical rules in the kb to create more natural questions that still require multi-hop reasoning. through experiments, we demonstrated that our dataset ensures multi-hop reasoning while being challenging for the multi-hop models. we also demonstrated that bootstrapping the multi-hop mrc dataset is beneficial by utilizing large-scale available data on wikipedia and wikidata.