Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering (2023)

Abstract: To explain the predicted answers and evaluate the reasoning abilities of models, several studies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA) datasets. However, it remains an open question as to how effective UR tasks are for the QA task when training models on both tasks in an end-to-end manner. In this study, we address this question by analyzing the effectiveness of UR tasks (including both sentence-level and entity-level tasks) in three aspects: (1) QA performance, (2) reasoning shortcuts, and (3) robustness. While the previous models have not been explicitly trained on an entity-level reasoning prediction task, we build a multi-task model that performs three tasks together: sentence-level supporting facts prediction, entity-level reasoning prediction, and answer prediction. Experimental results on 2WikiMultiHopQA and HotpotQA-small datasets reveal that (1) UR tasks can improve QA performance. Using four debiased datasets that are newly created, we demonstrate that (2) UR tasks are helpful in preventing reasoning shortcuts in the multi-hop QA task. However, we find that (3) UR tasks do not contribute to improving the robustness of the model on adversarial questions, such as sub-questions and inverted questions. We encourage future studies to investigate the effectiveness of entity-level reasoning in the form of natural language questions (e.g., sub-question forms).

## introduction the task of multi-hop question answering (qa) requires a model to read and aggregate information from multiple paragraphs to answer a given question (figure 1a ). several multi-hop qa datasets have been proposed, such as qangaroo @xcite , hotpotqa @xcite , and musique @xcite . in hotpotqa, the authors provide sentence-level supporting facts (sfs) to test the reasoning ability and explainability of the models. however, owing to the design of the sentence-level sfs task (binary classification) and the redundant information in the sentences, @xcite and @xcite show that the sentence-level sfs are insufficient to explain and evaluate multi-hop models in detail. to address this issue, r 4 c @xcite and @xcite datasets provide an entity-level reasoning prediction task to explain and evaluate the process of answering questions. entity-level reasoning information is defined as a set of triples that describes the reasoning path from question to answer (figure 1b ). several previous studies @xcite utilize sentence-level sfs and/or entity-level reasoning information to build explainable models by using question decomposition @xcite or predicting sentence-level sfs. the advantages of these pipeline models are that they can exploit the underlying reasoning (ur) process in qa and their predicted answers are more interpretable. however, the question remains as to how effective training on ur tasks is for the qa task in an end-to-end manner. although a few end-to-end models have also been introduced @xcite , these models are not explicitly trained on entity-level and answer prediction tasks. in addition to the triple form, the sub-question form is another way to utilize entity-level reasoning information. specifically, @xcite utilize question decomposition as an additional subquestion evaluation for bridge questions (there are two types of questions: bridge and comparison) in hotpotqa. they only use sub-questions for evaluation and do not fine-tune the models on them. in addition, @xcite use sub-questions for both evaluation and training. however, they only focus on comparison questions for date information. in contrast, we focus on the triple form of the entity-level information and conduct experiments using two datasets, 2wiki and hotpotqa-small (obtained by combining hotpotqa and r 4 c), which include both types of questions. in this study, we analyze the effectiveness of ur tasks (including both sentence-level and entitylevel) in three aspects: (1) qa performance, (2) reasoning shortcuts, and (3) robustness. first, qa performance is the final objective of the qa task. we aim to answer the following question: (rq1) can the ur tasks improve qa performance? for the second aspect, previous studies @xcite @xcite demonstrate that many questions in the multi-hop qa task contain biases and reasoning shortcuts @xcite , where the models can answer the questions by using heuristics. therefore, we aim to ask the following: (rq2) can the ur tasks prevent reasoning shortcuts? for the final aspect, to ensure safe development of nlp models, robustness is one of the important issues and has gained tremendous amount of research @xcite . in this study, we aim to test the robustness of the model by asking modified versions of questions, such as sub-questions and inverted questions. our question is (rq3) do the ur tasks make the models more robust? there are no existing end-to-end models that can perform three tasks simultaneously (sentence-level sfs prediction, entity-level prediction, and answer prediction); therefore, we first build a multi-hop bigbird-base model @xcite to perform these three tasks simultaneously. we then evaluate our model using two multi-hop datasets: 2wiki and hotpotqa-small. to investigate the ef-fectiveness of the ur tasks, for each dataset, we conduct three additional experiments in which the model is trained on: (1) answer prediction task, (2) answer prediction and sentence-level prediction tasks, and (3) answer prediction and entitylevel prediction tasks. we also create four debiased sets (figure 1c ) for 2wiki and hotpotqa-small for rq2. we create and reuse adversarial questions for 2wiki and hotpotqa-small for rq3. the experimental results indicate that the ur tasks can improve qa performance from 77.9 to 79.4 f1 for 2wiki and from 66.4 to 69.4 f1 for hotpotqa-small (rq1). the results of the models on the four debiased sets reveal that the ur tasks can be used to reduce reasoning shortcuts (rq2). specifically, when the model is trained on both answer prediction and ur tasks, the performance drop of the model on the debiased sets is lower than that when the model is trained only on answer prediction (e.g., 8.9% vs. 13.4% em). the results also suggest that the ur tasks do not make the model more robust on adversarial questions, such as sub-questions and inverted questions (rq3). our analysis shows that correct reconstruction of the entity-level reasoning task contributes to finding the correct answer in only 37.5% of cases. this implies that using entity-level reasoning information in the form of triples does not answer adversarial questions, in this case, the sub-questions. we encourage future work to discover the effectiveness of the entity-level reasoning task in the form of subquestions that have the same form as multi-hop qa questions. to evaluate the ur process of the models, derivation and evidence information were introduced in r 4 c and 2wiki, respectively. both derivation and evidence are sets of triples that represent the reasoning path from question to answer. the difference is the form; derivation in r 4 c uses a semi-structured natural language form, whereas evidence in 2wiki uses a structured form. we conduct experiments with both r 4 c (hotpotqa-small) and 2wiki. for consistency, we use the term entity-level reasoning prediction task to denote the derivation task in r 4 c and the evidence task in 2wiki. ## our multi-task model to investigate the usefulness of ur tasks for the qa task, we jointly train the corresponding tasks: sentence-level sfs prediction, entity-level prediction, and answer prediction. figure 3 illustrates our model. to handle long texts, we use the big-bird model @xcite , which is available in hugging face's transformers repository. 2 our model comprises three main steps: (1) paragraph selection, (2) context encoding, and (3) multi-task prediction. we use the named entity recognition (ner) models of spacy foot_2 and flair @xcite to extract all entities in the context and use them for the entity-level prediction task. paragraph selection following previous models @xcite @xcite , instead of using all the provided paragraphs, we first filter out answer-unrelated paragraphs. we follow the paragraph selection process described in @xcite . first, we retrieve first-hop paragraphs by using title matching or entity matching. we then retrieve second-hop paragraphs using the hyperlink information available in wikipedia. when we retrieve paragraphs, we reuse a paragraph ranker model foot_3 from the hierarchical graph network (hgn) model @xcite to rank input paragraphs using the probability of whether they contain sentence-level sfs. context encoding to obtain vector representations for sentences and entities, we first combine all the selected paragraphs into one long paragraph and then concatenate it with the question to form a context c. specifically, @xmath0, q 1 , ..., q m , [sep], p 1 , ..., p n , [sep]], where m and n are the lengths of the question q and the combined paragraph p (all selected paragraphs), respectively. the context c is then tokenized into l sub-words before feeding into bigbird to obtain the contextual representation c ′ of the sub-words: next, we obtain the representation s @xmath1, as follows: we also leverage the entity information to improve the contextual representation of sub-words c ′ as it is mainly used for the answer prediction task, which will be described in the next section. thus, the enhanced sub-word representation c ′′ i of the i-th sub-word is calculated as follows: ## datasets and evaluation metrics we mainly experiment with 2wiki and hotpotqasmall. we also train and evaluate our model on the full version of hotpotqa. we reuse and create debiased and adversarial sets for the evaluation. in hotpotqa. we obtain hotpotqa-small by combining hotpotqa @xcite with r 4 c. hotpotqa-small comprises three tasks as in 2wiki: ## results currently, there are no existing end-to-end models that explicitly train all three tasks together; therefore, in this study, we use our proposed model for analysis. we also compare our model with other previous models on the hotpotqa and 2wiki datasets. in general, the experimental results indicate that our model is comparable to previous models and can be used for further analyses. we focus more on the analysis; therefore, the detailed results of the comparison are presented in appendix b.3. ## conclusion we analyze the effectiveness of the underlying reasoning tasks using two multi-hop datasets: 2wiki and hotpotqa-small. the results reveal that the underlying reasoning tasks can improve qa performance. using four debiased sets, we demonstrate that the underlying reasoning tasks can reduce the reasoning shortcuts of the qa task. the results also reveal that the underlying reasoning tasks do not make the models more robust on adversarial examples, such as sub-questions and inverted questions. we encourage future studies to investigate the effectiveness of the entity-level reasoning task in the form of sub-questions.