Evidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering (2024)

Abstract: The long-standing goal of dense retrievers in abtractive open-domain question answering (ODQA) tasks is to learn to capture evidence passages among relevant passages for any given query, such that the reader produce factually correct outputs from evidence passages. One of the key challenge is the insufficient amount of training data with the supervision of the answerability of the passages. Recent studies rely on iterative pipelines to annotate answerability using signals from the reader, but their high computational costs hamper practical applications. In this paper, we instead focus on a data-driven approach and propose Evidentiality-Aware Dense Passage Retrieval (EADPR), which leverages synthetic distractor samples to learn to discriminate evidence passages from distractors. We conduct extensive experiments to validate the effectiveness of our proposed method on multiple abstractive ODQA tasks.

## introduction information retrieval (ir) has served as a core component in open-domain question answering (odqa) @xcite , which require the model to produce factually correct outputs based on a vast amount of knowledge in an unstructured text corpus. the predominant approach to odqa employs the simple yet effective retriever-reader framework @xcite , where the retriever (i.e., ir system) finds contexts that are relevant to the query from a large collection of texts, and the reader infers the final answer from the retrieved contexts. while augmenting the reader with a retriever is helpful when answerability aligns well with the relevance from the retriever, such an assumption does not always hold in abstractive odqa tasks, e.g., multi-hop qa @xcite , where target passages do not necessarily include the answer to the question. the misalignment between relevance and answerability in abstractive tasks poses a significant challenge to ir systems. the standard approach to building an ir system leverages human-annotated pairs of questions and relevant passages @xcite , but these ir datasets based on relevance provide only a weak supervision signal to abstractive tasks. this is particularly crucial for state-of-the-art ir systems, which train a dense passage retriever (dpr) @xcite ) using the relevance annotations to find relevant passages for a given question based on their learned vector representations. training a dense retriever with such misaligned supervision leads to suboptimal performance in abstractive tasks, as the retriever fails to capture evidence passages from the corpus based on answerablility @xcite . ## preliminaries a common approach to odqa tasks usually involves utilizing external knowledge from a large corpus of texts to produce factually correct outputs @xcite . due to the large search space in the corpus, a retriever is used in such settings to find subsets of relevant passages to questions for the expensive reader. the predominant approach to passage retrieval is dpr @xcite , which leverages the efficient dualencoder architecture denoted as [f q , f p ] to encode questions and passages into a learned embedding space. for a question-relevant passage pair (q i , p + i ) and a set of n negative passages p - j , dpr is trained to maximize the relevance measure (e.g., the vector similarity) between the question q i and its relevant passage p + i : previous studies on dense retrieval have presented some straightforward strategies to further enhance the performance of dpr. one such approach is negative sampling @xcite , which exploits multiple retrievers to collect informative negative samples. while earlier work uses lexical retrievers such as bm25 @xcite for negative sampling, recent studies find that sampling hard negatives from finetuned encoders @xcite leads to more informative hard negative @xcite . despite these efforts, it still remains a challenge to train a dense retriever to the abstractive tasks. the main obstacle arises from the lack of largescale data with strong annotations of evidentiality @xcite @xcite , i.e., whether each of the passages contains evidence needed to answer the questions. to address this issue, recent studies @xcite @xcite employ an iterative pipeline that annotates evidentiality of the passages using supervision signals from the reader. however, using these complex model-centric approaches requires a significant amount of computing resources, which obstruct their deployment in various scenarios @xcite @xcite . in this work, we instead study the validity of a data-centric approach to enhance the quality of ir datasets to obtain strong supervisions for passage retrieval from weak supervision in ir datasets. ## methodology our goal is to train a dense retriever capable of distinguishing evidence passages from distractor passages within a corpus. in this section, we propose evidentiality-aware dense passage retrieval (eadpr), a novel learning approach for dense retrieval where the learned representation is conditioned on evidence spans (i.e., positive) and invariant to evidentially-false contexts (i.e., negative). ## analysis and discussion answer awareness. to see how eadpr achieves such improvement, we conduct a finegrained analysis to measure the model's capability of capturing evidence spans. for this purpose, we introduce an additional analytic metric, named answer-awareness (aa) score, by measuring how frequently the model deems an answer-masked passage more relevant than its original passage. formally, given a held-out set of t pairs (q i , p + i ) with gold answer annotations, we construct answermasked passages p ′ i by removing exact answer spans from p + i . aa score of a retriever is then computed as the proportion of (q i , p + i , p ′ i ) triplets where relevance scores ⟨q i , p + i ⟩ are higher than the scores ⟨q i , p ′ to measure aa score, we reuse the 1,382 gold (q, p + , p ′ ) triplets from nq test set used in section 5. figure 4 compares the aa score of eadpr with dpr trained under the same conditions (i.e., training strategies). we first observe that aa scores of a vanilla dpr significantly fall behind the theoretical upper bound, which indicates that the relevance measurement learned in dpr may not effectively capture the evidentiality-awareness such that retrievers constantly rank positive passages higher than counterfactual passages. while common training strategies such as negative sampling lead to some increase in aa scores, there is still question: who died in the plane crash greys anatomy passage type text gold passage flight (grey's anatomy) ... american television medical drama grey 's anatomy . . . who are victims of an aviation accident fight to stay alive , but dr. lexie grey ( chyler leigh ) ultimately dies. ... dpr top-1 paul-louis halley. socata tbm 700 aircraft crash on 6 december 2003, during an approach to oxford airport. the plane went into an uncontrolled roll, killing halley, his wife, and the pilot. . . . dpr top-9 flight (grey's anatomy).. plane and awakens alone in the wood; his mangled hand having been pushed through the door of the plane. however, none are in as bad shape as lexie, who is crushed under ... eadpr top-1 comair flight 5191. after the crash to create an appropriate memorial for the victims, first responders, ... suffered serious injuries, including multiple broken bones, a collapsed lung, and severe bleeding. ... eadpr top-2 flight (grey's anatomy)... plane and awakens alone in the wood; his mangled hand having been pushed through the door of the plane. however, none are in as bad shape as lexie, who is crushed under ... in figure 4 , we further break down the the gold (q, p + , p ′ ) triplets with respect to their question types and measure aa scores of dpr and eadpr on subsets of test samples of different question types, i.e., who, when, what, where, how, and which. overall, we see that aa scores of dpr vary significantly across different question types, ranging from 65.07% to 80.68%. on the other hand, eadpr achieves significant improvements in aa scores for all question types and consistently shows better retrieval performance. among all question types, we see that dpr shows particularly low aa scores on who-, what-, and where-questions, whose answers tend to refer to named entities, i.e., names of people, locations, and objects. our hypothesis is that dpr often fails to identify the presence of target entities, which serve as causal features in evidence passages. while dpr is capable of retrieving passages with relevant semantics such as aircraft crash, it fails to identify key named entities in the question such as greys anatomy. in contrast, we observe eadpr ranks evidence passages with key entities higher than dpr (i.e., top-2 from eadpr compared to top-9 from dpr), suggesting that eadpr learns to differentiate evidence passages from their distractors in which key entities are absent. in some sense, our approach is in line with previous methods based on salient span masking @xcite , where the retriever is trained to predict masked salient spans with the help of a reader. robustness. we have assumed that eadpr learns to discriminate between evidence and distractor passages. to validate this assumption, we perform a simulation test in which we synthesize and add distractor passages into the corpus to measure the robustness of eadpr to these samples. this scenario is often encountered in real-world corpora, where there is a surplus of passages with similar contexts but lack definitive evidence @xcite @xcite . specifically, we create plausible distractor passages using a large language model (i.e., chatgpt). the model is prompted to generate near-duplicate samples for each query that mimic the context of evidence passage but leave out the key evidence. we collect these near-duplicates for 1,382 test queries from nq with annotated evidence passages and include at most 20 near-duplicates per query. figure 5 shows the performance of the dense retrievers on text corpora with varying number of near-duplicates. we observe that the retrieval performance (i.e., top-k accuracy) decreases substantially when given more near-duplicates, indicating that dense retrievers are vulnerable to the presence of distractor passages. on the other hand, we observe that eadpr is relatively robust against the effect from additional distractor samples, showing promise for robust passage retrieval on a noisy realworld corpus. ## conclusion in this work, we address the misalignment problem in dense retrievers for abstractive qa tasks, where relevance supervisions from ir datasets are not well-aligned with answerability of passages for questions. to overcome the abstractiveness of odqa tasks, we present eadpr, which augments distractor samples to train an evidentiality-aware retriever by learning to distinguish between evidence and distractor samples. our experiments show promising results in many odqa tasks, indicating that eadpr not only enhances model performance on both retrieval and downstream tasks but also improves robustness to distractors.