Counterfactual Multihop QA : A Cause-Effect Approach for Reducing Disconnected Reasoning (2023)

Abstract: Multi-hop QA requires reasoning over multiple supporting facts to answer the question. However, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred as disconnected reasoning problem. To alleviate this issue, we propose a novel counterfactual multihop QA, a causal-effect approach that enables to reduce the disconnected reasoning. It builds upon explicitly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect. With the causal graph, a counterfactual inference is proposed to disentangle the disconnected reasoning from the total causal effect, which provides us a new perspective and technology to learn a QA model that exploits the true multi-hop reasoning instead of shortcuts. Extensive experiments have been conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning. For example, our method achieves 5.8% higher points of its Supps score on HotpotQA through true multihop reasoning. The code is available at https://github.com/guowzh/CFMQA .

## introduction multi-hop question answering (qa) @xcite @xcite ) requires the model to reason over multiple supporting facts to correctly answer a complex question. it is a more challenging task than the single-hop qa since not only the correct answer but the explicit reasoning across multiple evidences should be provided. hence, recent work @xcite has shown that the multi-hop qa is always formulated as two sub-tasks: question * *corresponding author answering and support identification. for example, @xcite solved the multi-hop qa via 1) question answering which uses a bert @xcite span prediction model to answer the questions, and 2) support identification which aims to identify the supporting sentences. @xcite proposed pathfid based on fusionin-decoder (fid) @xcite for question answering and support identification. however, dividing the multi-hop qa into question answering and support identification doesn't mean that qa models answer the questions according to what the multi-hop qa wants. these qa models are all based on the large pre-training language models, e.g., bert @xcite and xlnet @xcite , these black-box language models are rather opaque models in terms of the reasoning processes. it may results in one main problem of multihop qa models: disconnected reasoning @xcite , which allows the qa models to exploit the reasoning shortcuts @xcite instead of multihop reasoning to cheat and obtain the right answer. taking fig. 1 as an example, to answer the question "until when in the u.s. senate", multi-hop qa requires to answer the question with a true reasoning path (e.g., the second @xmath0. 1 ). however, the black-box qa models can also infer the correct answer by just utilizing the types of problems, e.g., we can find the corresponding fact "from 2005 to 2008" in the contexts to answer this type of question "until when" without reasoning. to address the above problem, two issues should be considered: 1) how to measure the disconnected reasoning? to the best of our knowledge, @xcite firstly defined an evaluation measure, dire in short, to measure how much the qa model can cheat via disconnected reasoning. a probing dataset should be constructed to measure the dire. and 2) how to reduce the disconnected reasoning? one possible solution is to strengthen the training dataset via extra annotations or adversarial examples, which makes it cannot find the correct answers by only one supporting fact. for example, @xcite constructed the adversarial examples to generate better distractor facts. besides, counterfactual intervention @xcite had also been explored to change the distribution of the training dataset. however, when these existing approaches decrease the disconnected reasoning, the original performance also drops significantly. it is still challenging to reduce disconnected reasoning while maintaining the same accuracy on the original test set. motivated by causal inference @xcite @xcite , we utilize the counterfactual reasoning to reduce the disconnected reasoning in multi-hop qa and also obtain the robust performance on the original dataset. we formalize a causal graph to reflect the causal relationships between question (q), contexts, and answer (y ). to evaluate the disconnected reasoning, contexts are further divided into two subsets: s is a supporting fact and c are the remaining supporting facts. hence, we can formulate the disconnected reasoning as two natural direct causal effects of (q, s) and (q, c) on y as shown in fig. 1 . with the proposed causal graph, we can relieve the disconnected reasoning by disentangling the two natural direct effects and the true multihop reasoning from the total causal effect. a novel counterfactual multihop qa is proposed to disentangle them from the total causal effect. we utilize the generated probing dataset proposed by @xcite and dire to measure how much the proposed multi-hop qa model can reduce the disconnected reasoning. experiment results show that our approach can substantially decrease the disconnected reasoning while guaranteeing the strong performance on the original test set. the results indicate that the proposed approach can improve the true multi-hop reasoning capability. the main contribution of this paper is threefold. firstly, our counterfactual multi-hop qa model formulates disconnected reasoning as two direct causal effects on the answer, which is a new perspective and technology to learn true multi-hop reasoning. secondly, our approach achieves notable improvement on reducing disconnected reasoning compared to various baselines. thirdly, our causaleffect approach is model-agnostic and can be used for reducing disconnected reasoning in many multihop qa architectures. ## related work multi-hop question answering (qa) requires the model to retrieve the supporting facts to predict the answer. many approaches and datasets have been proposed to train qa systems. for example, hotpotqa @xcite dataset is a widely used dataset for multi-hop qa, which consists of fullwiki setting @xcite @xcite @xcite and distractor setting @xcite @xcite @xcite . in fullwiki setting, it first finds relevant facts from all wikipedia articles and then answers the multi-hop qa with the found facts. the retrieval model is important in this setting. for instance, smrs @xcite and dpr @xcite found the implicit importance of retrieving relevant information in the semantic space. entitycentric @xcite , cogqa @xcite and golden retriever @xcite explicitly used the entity that is mentioned or reformed in query key words to retrieve the next hop document. furthermore, pathretriever @xcite and hopretriever @xcite can iteratively select the documents to form a paragraph-level reason path using rnn. mdpr @xcite retrieved passages only using dense query vectors many times. these methods hardly discuss the qa model's disconnected reasoning problem. in the distractor setting, 10 paragraphs, two gold paragraphs and eight distractors, are given. many methods have been proposed to strengthen the model's capability of multi-hop reasoning, using graph neural network @xcite @xcite or adversarial examples or counterfactual examples @xcite or the sufficiency of the supporting evidences @xcite or make use of the pre-trained language models @xcite . however, @xcite demonstrated that many compositional questions in hotpotqa can be answered with a single hop. it means that qa models can take shortcuts instead of multi-hop reasoning to produce the corrected answer. to relieve the issue, @xcite added adversarial examples as hard distractors during training. re-cently, @xcite proposed an approach, dire, to measure the model's disconnected reasoning behavior and used the supporting sufficiency label to reduce the disconnected reasoning. @xcite selected the supporting evidence according to the sentence causality to the predicted answer, which guarantees the explainability of the behavior of the model. while the original performance also drops when reducing the disconnected reasoning. causal inference. recently, causal inference @xcite has been applied to many tasks of natural language processing and computer vision, and it shows promising results and provides strong interpretability and generalizability. the representative works include counterfactual intervention for visual attention @xcite , causal effect disentanglement for vqa @xcite , the back-door and frontdoor adjustments @xcite @xcite . our method can be viewed as a complement of the recent approaches that utilize the counterfactual inference @xcite to identify the supporting facts and predict the answer. ## preliminaries in this section, we use the theory of causal inference @xcite to formalize our multi-hop reasoning method. suppose that we have a multi-hop dataset d and each instance has the form of (q, p ; y ), where q is a question and @xmath1, s 2 , • • • , s n } is a context consisting of a set of n paragraphs. and y is the ground-truth label. given a question q with multiple paragraphs as a context p , the multi-hop qa models are required to identify which paragraphs are the supporting facts and predict an answer using the supporting facts. causal graph. in multi-hop qa, multiple supporting facts are required to predict the answer. while qa models may use only one fact to give the answer, which is referred to as disconnected reasoning. for example, given a question q and only a paragraph s, the disconnected reasoning model can predict the correct answer or correctly determine whether the paragraph s is the supporting fact. hence, to define the causal graph of disconnected reasoning, the context p is further divided into a paragraph s and the remaining paragraphs c. now the (q, the causal graph for multi-hop qa is shown in figure 1 (a), where nodes denote the variables and directed edges represent the causal-and-effect relationships between variables. the paths in figure 1 (a) are as follows. based on the above, the effect of q, s, c on y can be divided into: 1) shortcut impacts, e.g., (q, s) → k 1 → y and (q, c) → k 2 → y , and 2) reasoning impact, e.g., (q, s, c) → hence, to reduce the multi-hop qa model's disconnected reasoning proposed in @xcite , we should exclude shortcut impacts (k 1 → y and k 2 → y ) from the total effect. ## experiments we extensively conduct the experiments on the hot-potqa @xcite dataset in the distractor setting. see appendix a.2 for more experimental results on the other multihop benchmark 2wiki-multihopqa @xcite . metrics and dataset: to measure multi-hop qa models, we report two results: 1) original denotes the model's original performance (larger is better) and 2) dire denotes how much the qa model is cheated (smaller is better). please note original-dire denotes the model's true multi-hop reasoning. a probing dataset should be constructed to measure dire. to the best of our knowledge, only @xcite provides the probing dataset of the development set of hotpotqa. thus we use the development set of hotpotqa to evaluate the original and dire. specifically, the probing dataset for hotpotqa in the distractor setting divides each example of the original dataset into two instances, both of which only contain one of two ground truth supporting paragraphs respectively. if the multihop qa model can arrive at the correct test output on two instances, it means that the model performs disconnected reasoning on the original example. please refer to @xcite for more details. following the dire @xcite , we report the metrics for hotpotqa: answer span (ans), supporting paragraphs (supp p ), supporting sentences (supp s ), joint metrics (ans +supp p , ans+ supp s ). we show both em scores and f1 scores to compare the performance between baselines and our counterfactual multi-hop reasoning method. baselines: first, we simply use the bert @xcite to predict the answer, supporting sentences and supporting paragraphs as the baseline. bert + ours denotes that we apply our counterfactual multi-hop reasoning method based on bert as the backbone. the proposed approach is model-agnostic and we also implement it on several multi-hop qa architectures, including dfgn @xcite , hgn @xcite and xlnet in dire @xcite . our proposed algorithm also can be implemented on other baselines. ## conclusion in this work, we proposed a novel counterfactual reasoning approach to reduce disconnected reasoning in multi-hop qa. we used the causal graph to explain the existing multi-hop qa approaches' behaviors, which consist of the shortcut impacts and reasoning impacts. the shortcut impacts capture the disconnected reasoning and they are formulated as the natural direct causal effects. then we constructed the counterfactual examples during the training phase to estimate the both natural direct effects of question and context on answer prediction as well as supporting facts identification. the reasoning impact represents the multi-hop reasoning and is estimated by introducing learnable parameters. during the test phase, we exclude the natural direct effect and utilize the true multi-hop effect to decrease the disconnected reasoning. experimental results demonstrate that our proposed counterfactual reasoning method can significantly drop the disconnected reasoning on the probing dataset and guarantee the strong performance on the original dataset, which indicates the most progress of the multi-hop qa model is attributed to the true multi-hop reasoning. besides, our approach is modelagnostic, and can be applied to other multi-hop qa architectures to avoid exploiting the shortcuts.