Improving Evidence Retrieval on Claim Verification Pipeline through Question Enrichment (2024)

Abstract: The AVeriTeC shared task introduces a new real-word claim verification dataset, where a system is tasked to verify a real-world claim based on the evidence found in the internet.In this paper, we proposed a claim verification pipeline called QueenVer which consists of 2 modules, Evidence Retrieval and Claim Verification.Our pipeline collects pairs of <Question, Answer> as the evidence. Recognizing the pivotal role of question quality in the evidence efficacy, we proposed question enrichment to enhance the retrieved evidence. Specifically, we adopt three different Question Generation (QG) technique, muti-hop, single-hop, and Fact-checker style. For the claim verification module, we integrate an ensemble of multiple state-of-the-art LLM to enhance its robustness.Experiments show that QueenVC achieves 0.41, 0.29, and 0.42 on Q, Q+A, and AVeriTeC scores.

## introduction claim verification has become critical in the past few years due to the widespread of false information. this highlight the needs for robust automated system for claim verification. to advance the research area, benchmark datasets and challenges such as fever @xcite and fever-ous @xcite have been introduced and subsequent systems @xcite @xcite @xcite @xcite have demonstrated progress in claim verification. nevertheless, given the artificial claims and structured wikipedia evidence in fever and fever-ous, those systems have been optimized primarily under this condition. verifying real-world claim such as news claim still poses a significant challenge due to the complexity of sources, varying contexts, and the potential for misleading or evolving information. recently, a new claim verification benchmark on real-world called averitec @xcite was introduced. in this benchmark, the system is required to retrieve relevant document from articles across the internet and extract essential information from the articles that can debunk the claim. then, the system must classify the claim as supported, refuted, not enough evidence, or conflicting evidence/cherrypicking. compare with previous datasets that relies on synthetic claims derived from wikipedia, averitec focused on real-world claims. additionally, question-answer pairs have been introduced to capture reasoning steps and include annotations for conflicting evidences, offering a more nuanced approach to claim verification. in this dataset, question generation is a structured process aimed at deconstructing the reasoning used in fact-checking. annotators identify key aspects of a claim that require verification by reading original claim, relevant fact-checking source(s) and original source of the claim. they have been tasked to generate questions that would help break verification into the smaller steps. these questions need to be designed to extract specific pieces of evidences that would be required to verify claim. in this paper, we propose question enrichment claim verification (qecv) consisting of 2 modules, evidence retrieval and claim verification. to enhance the quality of the retrieved evidence, we adopt three different question generation approaches; multi-hop, single-hop, and fact-checker style. single-hop aims to retrieve more general evidence to verify the claim, while multi-hop targets more detailed evidence for each component of the claim. fact-checker style mimic how human factchecker generate questions by conditioning on both the claim and the content article. in contrast, singlehop and multi-hop solely rely on the claim for question generation. our claim verification module combine two different approaches: evidencelevel verifier and claim-level verifier. the former classify intermediate label to individual piece of evidence, which are subsequently aggregated to determine the claim label. conversely, the latter directly classifies the claim label based from all the retrieved evidence. to leverage the strength of each approach, we employ a voting-based ensemble model to aggregate the output and obtain the final label. our pipeline achieves 0.41, 0.29, and 0.42 on q, q+a, and averitec scores respectively, which outperforms the baseline model with a substantial margin. ## pipeline as shown in figure 1 , our pipeline consists of two modules: evidence retrieval and claim verification. the input claim first passes through our three variants of evidence retrieval to retrieve relevant pairs @xmath0, answer>. each variant generate questions from the claim and retrieve relevant articles through faiss: a library for efficient similarity search @xcite . then, it outputs list @xmath1, @xmath2. thereafter, the claim sentence and the retrieved evidence are fed to the claim verification module to predict the final label. the detail of each module will be elaborated in subsequent subsections. ## conclusion in this paper, we introduced the qecv, a pipeline for verifying real-world claims. improving the evidence retrieval through question enrichment enable the framework to cover more evidence for verifying the claim, thus achieves 0.41 and 0.30 for the q and q+a performance on the test set. additionally, our pipeline combines across various claim verifier variants and llms to leverage their unique strengths, resulting in more robust verification process and an 0.42 averitec score on the test set.