Modeling Multi-hop Question Answering as Single Sequence Prediction (2022)

Abstract: Fusion-in-decoder (Fid) (Izacard and Grave, 2020) is a generative question answering (QA) model that leverages passage retrieval with a pre-trained transformer and pushed the state of the art on single-hop QA. However, the complexity of multi-hop QA hinders the effectiveness of the generative QA approach. In this work, we propose a simple generative approach (PathFid) that extends the task beyond just answer generation by explicitly modeling the reasoning process to resolve the answer for multi-hop questions. By linearizing the hierarchical reasoning path of supporting passages, their key sentences, and finally the factoid answer, we cast the problem as a single sequence prediction task. To facilitate complex reasoning with multiple clues, we further extend the unified flat representation of multiple input documents by encoding cross-passage interactions. Our extensive experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets: HotpotQA and IIRC. Besides the performance gains, PathFid is more interpretable, which in turn yields answers that are more faithfully grounded to the supporting passages and facts compared to the baseline Fid model.

## introduction leveraging knowledge to make complex reasoning has been a fundamental problem of artificial intelligence. open-domain question answering (qa) @xcite is an integral part of such a line of research with impactful applications @xcite , where the task is to answer general domain questions by gathering evidence from a large collection of documents. while super-human level performance has been achieved on single-passage reading comprehension dataset like squad @xcite , open-domain qa still has a long way to go, especially for questions requiring more complex reasoning. the main challenge in the task of complex qa, namely multihop qa, is that it requires a qa system to combine multiple pieces of evidence from multiple documents @xcite @xcite . even for single-hop qa, it has been shown challenging for extractive qa models to effectively aggregate evidence from the combined pool of multiple passages, which has been the focus of recent work @xcite @xcite . recent work @xcite has demonstrated the promise of a generative approach at combining evidences from multiple passages for answer generation. thanks to large pre-trained transformers like t5 @xcite , @xcite introduced fusion-in-decoder (fid) that leverages passage retrieval with generative models for open-domain qa, achieving state-of-the-art scores across several single-hop qa benchmarks. however, we observe that the success of the fid model does not extend to multi-hop qa, which is corroborated by the findings in @xcite . further, the fid model is a rather opaque model in terms of interpretation of the answer generation process. this capability becomes especially important for multi-hop qa, which requires sequential reasoning across multiple evidences from the pool of retrieved passages. in this work, we propose pathfid, a generative qa model that learns to generate an answer along with a reasoning path to improve its capability of multi-hop reasoning. pathfid extends multi-hop qa beyond just answer generation by explicitly modeling the full reasoning path to resolve the answer with a generative sequence-tosequence model. to this end, we cast the problem as a single sequence prediction task that simultaneously models reasoning path consisting of supporting passages and facts, and eventually the factoid answer. furthermore, we extend pathfid to allow for cross-passage interactions between the figure 1 : an example of multi-hop question from hotpotqa dataset. it requires fusing multiple evidences (supporting facts) from multiple passages in a certain order to arrive at the correct answer. we formulate the entire problem as a single sequence prediction of the linearized hierarchical path ending with the answer. ## problem setup and background in this section, we formally introduce the problem setup and establish the necessary background. ## pathfid reader for multi-hop qa in this section, we introduce a generative reader (pathfid) for k-hop qa that jointly generates an alternating sequence of passage-level and factlevel clues on the reasoning path by more explicit fusion of evidence from the pool of input passages to arrive at the correct answer. ## conclusion in this work, we propose a generative question answering (qa) approach that models multi-hop qa as a single sequence prediction task. it learns to generate an answer along with a reasoning path to improve its capability of multi-hop reasoning. our experiments on prominent multi-hop qa benchmarks, hotpotqa and iirc, validate the promise and effectiveness of our proposed method path-fid and its extension pathfid+. future work will explore (1) our pathfid approach more closely with text retrieval models in open-domain qa scenarios and (2) more explicit grounding on the input information to make our approach even more interpretable and controllable. in figure 4 and 5, we visualize the correlation between supporting evidence and answer prediction performances for comparison and bridge question types, respectively. to obtain these plots, we first split the examples into 10 buckets where n-th bucket contains the examples with support-f1 score in (10 * (n -1), 10 * n] percentile for @xmath0, 2, . . . , 10}. then, we take the average answer prediction accuracy (both em and f1) over these examples for each bucket, and report this number on the y-axis of the plot at the corresponding support-f1 bucket on the x-axis, while dropping the empty buckets. note that @xmath1. also note that the size of a data point on the figure reflects the number of examples in the corresponding bucket as also indicated by the legend. from figures 4 and 5 , we can observe that the accuracy of the generated answers is significantly lower, 30% for bridge and 10% for comparison, for the first bucket with zero support-f1 compared to buckets with positive support-f1 score. this suggests that the model has a difficult time figuring out the an-swer when the supporting evidence prediction is poor. another observation that holds for both categories is the general trend of increased answer quality as the supporting fact prediction improves. combining these two points provide additional evidence (in addition to