If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering (2021)

Abstract: Multi-hop reasoning requires aggregation and inference from multiple facts. To retrieve such facts, we propose a simple approach that retrieves and reranks set of evidence facts jointly. Our approach first generates unsupervised clusters of sentences as candidate evidence by accounting links between sentences and coverage with the given query. Then, a RoBERTa-based reranker is trained to bring the most representative evidence cluster to the top. We specifically emphasize on the importance of retrieving evidence jointly by showing several comparative analyses to other methods that retrieve and rerank evidence sentences individually. First, we introduce several attention- and embedding-based analyses, which indicate that jointly retrieving and reranking approaches can learn compositional knowledge required for multi-hop reasoning. Second, our experiments show that jointly retrieving candidate evidence leads to substantially higher evidence retrieval performance when fed to the same supervised reranker. In particular, our joint retrieval and then reranking approach achieves new state-of-the-art evidence retrieval performance on two multi-hop question answering (QA) datasets: 30.5 Recall@2 on QASC, and 67.6% F1 on MultiRC. When the evidence text from our joint retrieval approach is fed to a RoBERTa-based answer selection classifier, we achieve new state-of-the-art QA performance on MultiRC and second best result on QASC.

## introduction recent advances in question answering (qa) have achieved excellent performance on several benchmark datasets @xcite , even when relying on partial @xcite , incorrect @xcite or no supporting knowledge @xcite . specifically, black-box neural qa methods have shown to rely on spurious signals confirming unfaithful or non-explainable behavior 1. rna is a small molecule that can squeeze through pores in the nuclear membrane 2. cells with a nuclear membrane are called eukaryotic. ## proposed approach we summarize the overall execution flow of our qa system in figure 2 . the four key components of the system are explained below. in its first step, wair uses a query that consists of the non-stop words of the original ques- rna is a small molecule that can squeeze through pores in || eukaryotic cells wair {sent 0,sent 1} {sent 0,sent 2} . .. {sent 8,sent 9} @xmath0, sent 1, sent 2} {sent 0, sent 1, sent 3} . . . . . . . . {sent 7,sent 8, sent 9} @xmath1..… membrane sent 1 -rna synthesis in eukaryotic cells …. sent 2 -cells with a nuclear membrane . sent n roberta reranker generating sets of evidence sentences answer selector qasc (mcqa) multirc (classification) initial evidence sentence retrieval reranking evidences using supervised roberta answer selection module -roberta generating candidate evidences the left branch implements a baseline method, which retrieves and feeds candidate evidence sentences to reranker individually. we denote this method "single sentence retrieval and reranking" (singlerr). the method on the right branch feeds wair candidate chains to the roberta reranker which jointly reranks the complete evidence text (referred to as jointrr). for both classifiers, we used roberta-base with a learning rate of 1e-5, maximum sequence length of 256 5 , batch size of 8, and 4 epochs. for the singlerr approach, all the evidence sentences having probability larger than 0.5 are concatenated to create the final evidence text. for jointrr approach, the evidence set with the highest regression score is selected. similarly, all the sentences in this set are concatenated into a single text. ## experimental results we focus on complex non-factoid and long answer span based explainable multi-hop datasets: multi-sentence reading comprehension (multirc): a reading comprehension dataset provided in the multiple-choice qa format @xcite . every question is supported by one document, from which the answer and justification sentences must be extracted. wair retrieves @xmath2, foot_4 which are separately considered as candidates in the downstream components of singlerr. for the jointrr approach, we generate combinations of evidence texts with @xmath3, 3, 4} sentences, i.e., @xmath4,3,4} . we use the original multirc dataset 9 which includes the gold annotations for evidence text. question answering using sentence composition (qasc): a multiple-choice qa dataset @xcite , where each question is provided with 8 answer candidates, out of which 4 candidates are hard adversarial choices. the evidence sentences are to be retrieved from a large kb of 17.2 million facts. similar to @xcite , wair first retrieves @xmath5, where the query concatenates the question and candidate answer texts. wair uses each of these retrieved sentences to reformulate and reweigh the query, to retrieve an additional 1 sentence in a second iteration. this results in a total of 20 candidate evidence sentences for a given question and candidate answer. we generate evidence chains using the same approach as the one used for multirc, except here we focus on @xmath6, i.e., @xmath7, because all questions in qasc are annotated with only two gold justification sentences. we report qa and evidence selection performances in both the datasets using standard evaluation measures @xcite . ## conclusion we introduced a simple unsupervised approach for retrieving candidate evidence chains that after reranking achieves state-of-the-art evidence retrieval performance on two multi-hop qa datasets: qasc and multirc. we highlight the importance of generating and feeding candidate evidence chains by showing several benefits over the widely followed approach that retrieves evidence sentences individually. further, we introduced few attention and embedding analyses demonstrating that jointly retrieving and reranking chains assist in learning compositional information, which is also beneficial to the downstream qa task. overall, our work highlights the strengths and potential of joint retrieval+reranking approaches for future works.