Exploring Retrieval Augmented Generation For Real-world Claim Verification (2024)

Abstract: Automated Fact-Checking (AFC) has recently gained considerable attention to address the increasing misinformation spreading in the web and social media. The recently introduced AVeriTeC dataset alleviates some limitations of existing AFC benchmarks. In this paper, we propose to explore Retrieval Augmented Generation (RAG) and describe the system (UPS participant) we implemented to solve the AVeriTeC shared task.Our end-to-end system integrates retrieval and generation in a joint training setup to enhance evidence retrieval and question generation. Our system operates as follows: First, we conduct dense retrieval of evidence by encoding candidate evidence sentences from the provided knowledge store documents. Next, we perform a secondary retrieval of question-answer pairs from the training set, encoding these into dense vectors to support question generation with relevant in-context examples. During training, the question generator is optimized to generate questions based on retrieved or gold evidence. In preliminary automatic evaluation, our system achieved respectively 0.198 and 0.210 AVeriTeC scores on the dev and test sets.

## introduction with the unprecedented growing of fake news in the web and on social media, several research efforts have been supported in the recent years to combat online misinformation. while manual factchecking is the most reliable method for verifying information, the large-scale amount of daily published and shared content has made the development of automated fact-checking solutions crucial to assist in the manual fact checking process. following such initiatives, the recently introduced averitec (automated verification of textual claims) dataset @xcite contributes to address the aforementioned challenges, and serves as a benchmark for the averitec shared task. in this paper, we report our findings in addressing the averitec shared task and describe the proposed system which is evaluated on its ability of verifying real-world claims with evidence from the web. in contrast to other fact-checking datasets such as fever @xcite , vitam-inc @xcite and feverous @xcite , averitec focuses on realistic scenarios where real-world claims are derived from the web rather than wikipedia. in this context, systems are required to retrieve evidence that either supports or refutes a given claim, using sources from either the web or a document collection scrapped from the web and provided by the organizers. based on this evidence, systems must categorize the claim as supported, refuted, not enough evidence (when there is insufficient evidence to make a determination), or conflicting evidence/cherry-picking (when both supporting and refuting evidence are present). a response is considered correct only if it includes both the accurate label and sufficient supporting evidence. due to the complexity of evaluating evidence retrieval automatically, a manual evaluation process will be conducted to ensure a fair assessment of the participant systems. ## averitec baseline the averitec shared task organizers proposed a pipeline system which comprises the following steps: 1) given a claim c, it is used as a query input of a search engine (google api) to obtain relevant urls which are parsed into sentences. the collection of sentences serves for evidence retrieval. 2) for each claim c, only the top 100 sentences {s i } 100 ## proposed system following the baseline pipeline, we propose a simpler end-to-end integrated system ( see figure 1 ) which relies on the retrieval augmented generation (rag) framework to solve the averitec challenge where retrieval and generation complement each other using joint training. at the first stage, we perform evidence dense retrieval after encoding all potential evidence sentences retrieved from the provided knowledge store documents. then, we perform a second retrieval of question-answer pairs from the training set (encoder into dense vectors) to support question generation with in-context examples. during training, the question generator learns to generate question given retrieved/gold evidence by jointly updating both generator and evidence/answer encoder using the rag loss @xcite . finally, a veracity prediction model is employed to label the retrieved evidence. ## results at inference time, we achieved the best performance with the number of retrieved evidence @xmath0, while higher values decreases both evidence retrieval and veracity checking. regarding the number of retrieved in-context examples l, we found that building the prompt using only @xmath1. we assume that our bertbase retrieval provides more useful in-context examples in the top retrieved qa pairs and does not need to re-rank evidence compared to the baseline model which relies on bm25 to retrieve evidence. indeed, while we do not perform evidence retrieval during training, we still update the bert retrieval encoder parameters using the claim-evidence similarity scores with the rag loss. this latter allows to learn retrieving more relevant evidence for the target question using the feedback from the question generator. we reports in ## conclusion we presented in this paper our participant system (ups) at the averitec shared task on verifying realworld claims with evidence from the web. in preliminary automatic evaluation, our system achieved respectively 0.198 and 0.210 averitec scores on the dev and test sets, and was ranked 13 out of 23 participant teams. in terms of limitations, our proposed system relies solely on the averitec training set which is relatively small size. we believe that our rag approach would benefit from more training data. moreover, experimenting with larger generator models may improve the quality of generated questions and thus the overall fact-checking performance.