Evidence Retrieval for Fact Verification using Multi-stage Reranking (2024)

Abstract: In the fact verification domain, the accuracy and efficiency of evidence retrieval are paramount. This paper presents a novel approach to enhance the fact verification process through a Multi-stage ReRanking (M-ReRank) paradigm, which addresses the inherent limitations of single-stage evidence extraction. Our methodology leverages the strengths of advanced reranking techniques, including dense retrieval models and list-aware rerankers, to optimise the retrieval and ranking of evidence of both structured and unstructured types. We demonstrate that our approach significantly outperforms previous state-of-the-art models, achieving a recall rate of 93.63% for Wikipedia pages. The proposed system not only improves the retrieval of relevant sentences and table cells but also enhances the overall verification accuracy. Through extensive experimentation on the FEVEROUS dataset, we show that our M-ReRank pipeline achieves substantial improvements in evidence extraction, particularly increasing the recall of sentences by 7.85%, tables by 8.29% and cells by 3% compared to the current state-of-the-art on the development set.

## introduction the proliferation of false and misleading information, fuelled by the rapid progress in artificial intelligence (ai), poses a significant societal threat, as highlighted in the world economic forum's report @xcite . for example, the widespread mis/dis-information about covid-19 vaccines has caused a surge in anti-vaccination sentiment online, leading to a low vaccination coverage @xcite . a recent study shows that low-veracity media-induced overconfidence exacerbates the adverse effects of widespread misinformation (i.e., fake news), especially in current global election scenarios @xcite . to combat this, researchers are focusing on developing automatic fact verification systems to prevent disinformation from spreading online @xcite . ## our approach the aim of the feverous open-domain fact verification @xcite benchmark's task is to verify a claim c based on content from wikipedia. we follow the widely adopted threestep pipeline, which involves (i) retrieving relevant pages from the wikipedia dump, (ii) extracting sentences and table cells as evidence from these pages, and (iii) predicting the veracity label of the given claim based on the compiled evidence set. in this work, we explore improving the first and second steps-wiki-page retrieval and evidence extraction-by employing our multi-stage retrieval pipeline. in the three-step pipeline, as shown in figure 2 , the wikipedia pages are first retrieved and refined by our m-rerank approach. the top five pages are then used to extract evidence of both formats (e.g., sentences and tables) in the second step. we train the models in the m-rerank pipeline separately for page, sentence and table retrieval. combining the first five sentences and five tables, we use see-st's @xcite cell-retriever to extract potential cell evidence. finally, at the third step, verification, we utilise dcuf @xcite , a method that converts evidence into dual-channel encodings to verify the claim. ## conclusion in this paper, we presented m-rerank, a multistage reranking framework designed to enhance the evidence retrieval process for fact verification tasks. our experiments with the feverous dataset demonstrate that m-rerank significantly improves the recall of evidence extraction, achieving a feverous-score jump of 10.84%/2.38% on development/test data compared to previous state-of-the-art methods. m-rerank pipeline is comprised of a sequence rerankers, e.g. cross-encoder/see-st, hybrank, hlatr. by leveraging the contextual interactions among multiple evidence pieces and incorporating both lexical and semantic similarities, m-rerank effectively addresses the challenges of retrieving relevant evidence in both unstructured formats, e.g. sentences, and structured formats, e.g. tables or cells. our ablation studies further validate the efficacy and contribution of each reranking stage, showcasing the robustness and adaptability of our approach. overall, m-rerank sets a new benchmark in the domain of fact verification, paving the way for more accurate and reliable verification systems. future work could explore more efficient multistage reranking architectures that reduce computational overhead while maintaining accuracy. this might involve developing lightweight rerankers or incorporating parallel processing techniques @xcite . moreover, further investigation is needed into handling the underrepresented nei category, possibly by adopting advanced data augmentation strategies or rebalancing techniques to improve the model's performance on rare labels @xcite . additionally, expanding the model's capacity to better utilise inter-evidence interactions in more complex evidence structures, such as multimodal data, would also be a valuable direction.