From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification (2023)

Abstract: Retrieval-enhanced methods have become a primary approach in fact verification (FV); it requires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-shelf retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence. We introduce the \textbf{feedback-based evidence retriever} (FER) that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. As a feedback signal we use the divergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence to produce the final claim label. Empirical studies demonstrate the superiority of FER over prevailing baselines.

## introduction the risk of misinformation has increased the demand for fact-checking, i.e., automatically assessing the truthfulness of textual claims using trustworthy corpora, e.g., wikipedia. existing work on fact verification (fv) commonly adopts a retrievalenhanced verification framework: an evidence retriever is employed to query the background corpus for relevant sentences, to serve as evidence for the subsequent claim verifier. high-quality evidence is the foundation of claim verification. currently, prevailing approaches to identifying high-quality evidence typically adopt off-the-shelf retrieval models from the information retrieval (ir) field for evidence retrieval @xcite @xcite . these models are usually based on the probability ranking principle (prp) @xcite , ranking sentences based on their likelihood of being relevant to the claim. however, the sentences retrieved in this way are assumed to be consumed by humans, and this may not align with a retrieval-enhanced verification framework @xcite : the top-ranked sentences produced by existing retrieval models do not always align with the judgments made by the claim verifier regarding what counts as evidence. we argue that when designing evidence retrieval models for fv, the notion of relevance should be reconceptualized as the utility that the verifier derives from consuming the evidence provided by the retrieval model, a viewpoint that aligns well with task-based perspectives on ir @xcite . hence, we assume that when training evidence retrievers, it is advantageous to obtain feedback from the claim verifier as a signal to optimize the retrieval process. therefore, we propose a shift in emphasis from relevance to utility in evidence retrieval for fv. we introduce the feedback-based evidence retriever (fer) that incorporates a feedback mechanism from the verifier to enhance the retrieval process. fer leverages a coarse-to-fine strategy. initially, it identifies a candidate set of relevant sentences to the given claim from the large-scale corpus. subsequently, the evidence retriever is trained using feedback from the verifier, enabling a re-evaluation of evidence within the candidate set. here, feedback is defined as the utility divergence observed when the verifier evaluates the sentences returned by the retriever, compared to when it consumes the ground-truth evidence for predicting the claim label. by measuring the utility criterion between the two scenarios, we can optimize the retriever specifically for claim verification. experimental results on the large-scale fact extraction and verification (fever) dataset @xcite demonstrate a 23.7% f1 performance gain over the sota baseline. ## background for the fever task @xcite , automatic fv systems need to determine a threeway veracity label, i.e., supports, refutes or not enough info, for each human-generated claim based on a set of supporting evidence from wikipedia. the fv system can be split into document retrieval, evidence retrieval, and claim verification phases. the first phase is to retrieve related documents from the corpus. the second phase aims to extract evidence from all sentences of the recalled documents. the final phase aggregates the information of the retrieved evidence to predict a claim label. in this work, our contributions are focused on the evidence retrieval step. the related work and more extensive discussions are included in appendix a.1. ## our approach based on the related documents (i.e., wikipedia pages) obtained by a common document retrieval method with entity linking @xcite , fer comprises two components: (i) coarse retrieval: using a prp-based retrieval model to recall a set of candidate sentences to the claim from the related documents; and (ii) fine-grained retrieval: selecting evidential sentences from the candidate set based on the feedback from the claim verifier. the overall architecture of fer is illustrated in figure 1 . ## conclusion in this paper, we proposed fer (feedback-based evidence retrieval) for fv, a novel evidence retrieval framework that incorporates feedback from the claim verifier. unlike traditional approaches based on prp, fer places an emphasis on the utility of evidence that contributes meaningfully to the verification process, going beyond mere relevance. through the integration of feedback from the verifier, fer effectively identifies the evidence that is both relevant and useful for claim verification. the experimental results on the fever dataset demonstrated the effectiveness of fer.