How Well Do Multi-hop Reading Comprehension Models Understand Date Information? (2022)

Abstract: Several multi-hop reading comprehension datasets have been proposed to resolve the issue of reasoning shortcuts by which questions can be answered without performing multi-hop reasoning. However, the ability of multi-hop models to perform step-by-step reasoning when finding an answer to a comparison question remains unclear. It is also unclear how questions about the internal reasoning process are useful for training and evaluating question-answering (QA) systems. To evaluate the model precisely in a hierarchical manner, we first propose a dataset, HieraDate , with three probing tasks in addition to the main question: extraction, reasoning, and robustness. Our dataset is created by enhancing two previous multi-hop datasets, HotpotQA and 2WikiMultiHopQA, focusing on multi-hop questions on date information that involve both comparison and numerical reasoning. We then evaluate the ability of existing models to understand date information. Our experimental results reveal that the multi-hop models do not have the ability to subtract two dates even when they perform well in date comparison and number subtraction tasks. Other results reveal that our probing questions can help to improve the performance of the models (e.g., by +10.3 F1) on the main QA task and our dataset can be used for data augmentation to improve the robustness of the models.

## introduction multi-hop reading comprehension (rc) requires a model to read and aggregate information from multiple paragraphs to answer a given question @xcite . several datasets have been proposed for this task, such as hotpotqa @xcite and 2wikimultihopqa (2wiki; @xcite . although the proposed models show promising performances, previous studies @xcite @xcite have demonstrated that existing multi-hop datasets contain reasoning shortcuts, in which the model can answer the question without performing multi-hop reasoning. there are two main types of questions in the previous multi-hop datasets: bridge and comparison. @xcite explored sub-questions in the question answering (qa) process for model evaluation. however, they only used the bridge questions in hotpotqa and did not fine-tune the previous multi-hop models on their dataset when performing the evaluation. therefore, it is still unclear about the ability of multi-hop models to perform step-by-step reasoning when finding an answer to a comparison question. hotpotqa provides sentence-level supporting facts (sfs) to explain the answer. however, as discussed in @xcite and @xcite , the sentence-level sfs cannot fully evaluate the reasoning ability of the models; to solve this issue, in addition to sentence-level sfs, these studies pro-vide a set of triples as the evidence information. for example, for the question in figure 1 , the evidence regards the dates of birth and death of two people, e.g., (maceo, date of death, july 4, 2001). we argue that simply requiring the models to detect a set of triples, in this case, cannot explain the answer to the question and cannot describe the full path from the question to the answer; additional operations, including calculations and comparisons, need to be performed to obtain the final answer. to deal with this issue, we introduce a dataset, hieradate, foot_0 consisting of the three probing tasks. our dataset is created by extending two existing multi-hop datasets, hotpotqa and 2wiki. as the first step of the proof of concept, we start with the date information through comparison questions because this information is available and straightforward to handle. moreover, based on the classification of comparison questions in @xcite , all comparison questions on date information require multi-hop reasoning for answering. we then use our dataset to evaluate two leading models, hgn @xcite and numnet+ @xcite on two settings: with and without fine-tuning on our dataset. we also conduct experiments to investigate whether our probing questions are useful for improving qa performance and whether our dataset can be used for data augmentation. our experimental results reveal that existing multi-hop models perform well in the extraction and robustness tasks but fail in the reasoning task when the models are not fine-tuned on our dataset. we observe that with fine-tuning, hgn can perform well in the comparison reasoning task; meanwhile, numnet+ struggles with subtracting two dates, although it can subtract two numbers. our analysis shows that questions that require both nu-merical and comparison reasoning are more difficult than questions that require only comparison reasoning. we also find that training with our probing questions boosts qa performance in our dataset, showing improvement from 77.1 to 82.7 f1 in hgn and from 84.6 to 94.9 f1 in numnet+. moreover, our dataset can be used as augmentation data for hotpotqa, 2wiki, and drop @xcite , which contributes to improving the robustness of the models trained on these datasets. our results suggest that a more complete evaluation of the reasoning path may be necessary for better understanding of multi-hop models' behavior. we encourage future research to integrate our probing questions when training and evaluating the models. ## related work in addition to @xcite , @xcite and @xcite are similar to our study. @xcite evaluated the previous models on the drop dataset to test their numerical reasoning ability. however, they did not investigate the internal reasoning processes of those models. @xcite proposed a framework for creating new examples using the perturbation of the reasoning path. our work is different in that their focus was on creating a framework, and it does not necessarily ensure the quality of all generated perturbation samples. moreover, we investigate the qa process in-depth, @xcite do not include all detailed questions (e.g., they do not include extraction task and comparison reasoning questions in figure 1 ). ## dataset construction our dataset is generated by using the two existing multi-hop datasets, hotpotqa and 2wiki (more details are in appendix b.1). ## experiments to comprehensively evaluate the top-performing multi-hop models, we conducted various experiments, including both with and without fine-tuning on our dataset. in addition, to discover the effectiveness of our dataset, we examine the usefulness of our probing tasks and investigate whether our dataset can be used for data augmentation. ## conclusion we proposed a new multi-hop rc dataset for comprehensively evaluating the ability of existing models to understand date information. we evaluated the top-performing models on our dataset. the results revealed that the models may not possess the ability to subtract two dates even when fine-tuned on our dataset. we also found that our probing questions could help to improve qa performance, and can be used for data augmentation. for future work, we will use the hierarchical manner in our dataset to apply to other types of questions such as numerical reasoning questions in drop. for diverse, explainable multi-hop question answering. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369-2380, brussels, belgium. association for computational linguistics.