Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering (2020)

Abstract: Evidence retrieval is a critical stage of question answering (QA), necessary not only to improve performance, but also to explain the decisions of the QA method. We introduce a simple, fast, and unsupervised iterative evidence retrieval method, which relies on three ideas: (a) an unsupervised alignment approach to soft-align questions and answers with justification sentences using only GloVe embeddings, (b) an iterative process that reformulates queries focusing on terms that are not covered by existing justifications, which (c) stops when the terms in the given question and candidate answers are covered by the retrieved justifications. Despite its simplicity, our approach outperforms all the previous methods (including supervised methods) on the evidence selection task on two datasets: MultiRC and QASC. When these evidence sentences are fed into a RoBERTa answer classification component, we achieve state-of-the-art QA performance on these two datasets.

## introduction explainability in machine learning (ml) remains a critical unsolved challenge that slows the adoption of ml in real-world applications @xcite @xcite . question answering (qa) is one of the challenging natural language processing (nlp) tasks that benefits from explainability. in particular, multihop qa requires the aggregation of multiple evidence facts in order to answer complex natural language questions @xcite . several multi-hop qa datasets have been proposed recently @xcite @xcite @xcite @xcite @xcite . while several neural methods have achieved state-of-theart results on these datasets @xcite @xcite , we argue that many of these directions lack a human-understandable explanation of their inference process, which is necessary to transition these approaches into realworld applications. this is especially critical for multi-hop, multiple choice qa (mcqa) where: (a) the answer text may not come from an actual knowledge base passage, and (b) reasoning is required to link the candidate answers to the given question @xcite . figure 1 shows one such multi-hop example from a mcqa dataset. in this paper we introduce a simple alignmentbased iterative retriever (air) foot_0 , which retrieves high-quality evidence sentences from unstructured knowledge bases. we demonstrate that these evidence sentences are useful not only to explain the required reasoning steps that answer a question, but they also considerably improve the performance of the qa system itself. unlike several previous works that depend on supervised methods for the retrieval of justification sentences (deployed mostly in settings that rely on small sets of candidate texts, e.g., hotpotqa, multirc), air is completely unsupervised and scales easily from qa tasks that use small sets of candidate evidence texts to ones that rely on large knowledge bases (e.g., qasc @xcite ). air retrieves justification sentences through a simple iterative process. in each iteration, air uses an alignment model to find justification sentences that are closest in embedding space to the current query @xcite , which is initialized with the question and candidate answer text. after each iteration, air adjusts its query to focus on the missing information @xcite in the current set of justifications. air also conditionally expands the query using the justifications retrieved in the previous steps. in particular, our key contributions are: our work falls under the revitalized direction that focuses on the interpretability of qa systems, where the machine's inference process is explained to the end user in natural language evidence text @xcite @xcite @xcite . several 2 in settings where external labeled resources are not used. question: exposure to oxygen and water can cause iron to (a) decrease strength (b) melt (c) uncontrollable burning (d) thermal expansion (e) turn orange on the surface (f) vibrate (g) extremes of temperature (h) levitate gold justification sentences: parallel evidence chain 1: qa approaches that focus on interpretability can be broadly classified into three main categories: supervised, which require annotated justifications at training time, latent, which extract justification sentences through latent variable methods driven by answer quality, and, lastly, unsupervised ones, which use unsupervised algorithms for evidence extraction. in the first class of supervised approaches, a supervised classifier is normally trained to identify correct justification sentences driven by a query @xcite @xcite . many systems tend to utilize a multi-task learning setting to learn both answer extraction and justification selection with the same network @xcite . although these approaches have achieved impressive performance, they rely on annotated justification sentences, which may not be always available. few approaches have used distant supervision methods @xcite to create noisy training data for evidence retrieval but these usually underperform due to noisy labels. in the latent approaches for selecting justifica-tions, reinforcement learning @xcite and pagerank @xcite have been widely used to select justification sentences without explicit training data. while these directions do not require annotated justifications, they tend to need large amounts of question/correct answer pairs to facilitate the identification of latent justifications. in unsupervised approaches, many qa systems have relied on structured knowledge base (kb) qa. for example, several previous works have used conceptnet @xcite to keep the qa process interpretable @xcite . however, the construction of such structured knowledge bases is expensive, and may need frequent updates. instead, in this work we focus on justification selection from textual (or unstructured) kbs, which are inexpensive to build and can be applied in several domains. in the same category of unsupervised approaches, conventional information retrieval (ir) methods such as bm25 @xcite have also been widely used to retrieve independent individual sentences. as shown by @xcite , and our recently, many supervised iterative justification retrieval approaches for qa have been proposed @xcite @xcite . while these were shown to achieve good evidence selection performance for complex questions when compared to earlier approaches that relied on just the original query @xcite , they all require supervision. as opposed to all these iterative-retrieval methods and previously discussed directions, our proposed approach air is completely unsupervised, i.e., it does not require annotated justifications. further, unlike many of the supervised iterative approaches @xcite ) that perform query reformulation in a continuous representation space, air employs a simpler and more interpretable query reformulation strategy that relies on explicit terms from the query and the previously retrieved justification. lastly, none of the previous iterative retrieval approaches address the problem of semantic drift, whereas air accounts for drift by controlling the query reformulation as explained in section 3.1. ## approach as shown in fig. 2 , the proposed qa approach consists of two components: (a) an unsupervised, iterative component that retrieves chains of justification sentences given a query; and (b) an answer classification component that classifies a candidate answer as correct or not, given the original question and the previously retrieved justifications. we detail these components in the next two sub-sections. ## experiments we evaluated our approach on two datasets: multi-sentence reading comprehension (mul-tirc), which is a reading comprehension dataset provided in the form of multiple-choice qa task @xcite . every question is based on a paragraph, which contains the gold justification sentences for each question. we use every sentence of the paragraph as candidate justifications for a given question. here we use the original justification sentences are to be retrieved from a kb having 17.2 million facts. as shown in the example of fig. 1 and also highlighted by @xcite , multiple evidence text are possible for a given question in qasc where the annotated gold justification sentences explain it more precisely. ## analysis to further understand the retrieval process of air we implemented several analyses. ## conclusion we introduced a simple, unsupervised approach for evidence retrieval for question answering. our approach combines three ideas: (a) an unsupervised alignment approach to soft-align questions and answers with justification sentences using glove embeddings, (b) an iterative process that reformulates queries focusing on terms that are not covered by existing justifications, and (c) a simple stopping condition that concludes the iterative process when all terms in the given question and candidate answers are covered by the retrieved justifications. overall, despite its simplicity, unsupervised nature, and its sole reliance on glove embeddings, our approach outperforms all previous methods (including supervised ones) on the evidence selection task on two datasets: multirc and qasc. when these evidence sentences are fed into a roberta answer classification component, we achieve the best qa performance on these two datasets. further, we show that considerable improvements can be obtained by aggregating knowledge from parallel evidence chains retrieved by our method. in addition of improving qa, we hypothesize that these simple unsupervised components of air will benefit future work on supervised neural iterative retrieval approaches by improving their query reformulation algorithms and termination criteria.