UHH at AV eri T e C : RAG for Fact-Checking with Real-World Claims (2024)

Abstract: This paper presents UHH’s approach developed for the AVeriTeC shared task. The goal of the challenge is to verify given real-world claims with evidences from the Web. In this shared task, we investigate a Retrieval-Augmented Generation (RAG) model, which mainly contains retrieval, generation, and augmentation components. We start with the selection of the top 10k evidences via BM25 scores, and continue with two approaches to retrieve the most similar evidences: (1) to retrieve top 10 evidences through vector similarity, generate questions for them, and rerank them or (2) to generate questions for the claim and retrieve the most similar evidence, again, through vector similarity. After retrieving the top evidences, a Large Language Model (LLM) is prompted using the claim along with either all evidences or individual evidence to predict the label. Our system submission, \textbf{UHH} , using the first approach and individual evidence prompts, ranks 6th out of 23 systems.

## introduction fact-checking is a process to (automatically) assess the truthfulness of a claim, which is an important task for some domains, e.g. journalism @xcite @xcite . the averitec shared task 1 @xcite aims at dealing with the challenge of verifying realworld claims with pieces of evidence from the web, as shown in figure 1 . recently, retrieval-augmented generation (rag) provides a remedy for some issues of large language models (llms), e.g. hallucination, while increasing the performance of especially knowledge-intensive tasks, including fact-checking @xcite . motivated by this, we investigate how to effectively leverage such a method in this shared task. our submission's pipeline is as follows; evidences (in the form of short texts like sentences foot_0 ) per claim provided by task organizers are ranked using bm25 @xcite and the top 10k evidences are selected. for retrieving the most relevant evidences, we consider two approaches: (1) retrieve-question: retrieving the most similar 10 evidences using vector similarity and generating questions for these evidences. then, evidences are reranked again based on vector similarity with evidences in the form of questionanswer.; (2) question-retrieve: generating questions for a claim, inspired from @xcite , where they see an improvement for the retrieval with decomposed questions. we retrieve the singlebest evidence per a question using vector similarity. the two approaches perform competitively in the development set. in the last step, we prompt llm with the retrieved evidences to predict the label. we experiment to prompt with either all evidences or one evidence at a time. in our experiments, prompting with individual evidence can reach higher scores. note that our pipeline resembles the steps conducted in the organizer's baseline @xcite , especially in the retrieve-question approach, for more details see section 4. the contributions of this paper as follows: retrieve top 10 evidences via vector similarity generate 3 questions for given claim generate questions for these evidences retrieve top evidence via vector similarity re-rank via vector similarity llm approach-1: retrieve -question approach-2: question -retrieve supported refuted not enough evidence conflicting evidence/cherry-picking sentences sentences evidences for claim claim select top 10k evidences via bm25 checking task with real-world claims and evidences from the web. considering the fact that our method is highly similar to the baseline, we also provide a list of main differences and/or improvements: our code 3 is publicly available. the remainder of the paper is structured as follows. we continue with the background, and then the methodology is explained in detail. in subsequent sections, we present the experimental setup and discuss the results. and finally, conclusions, future work, and limitations are discussed. fact-checking it is a challenging task to automate a fact-checking process @xcite , with different issues, for example, chen et al. (2022) discuss the challenges of complex political claims. many datasets have been developed for this task, e.g. the fever @xcite dataset from wikipedia sources. in the averitec shared task, the dataset contains real-world claims, as shown in figure 1 , annotated with question-answer pairs. ## methodology overview the pipeline used in our solution is shown in figure 2 . evidences per claim provided by task organizers are first ranked using bm25 @xcite . the highestranked 10k evidences to an input claim are selected. we have experimented two approaches to select the most similar evidences: (1) retrieving top 10 evidences first and then generating questions from evidences (retrieve-question), or (2) generating questions for a claim and retrieving the most similar evidence per question (question-retrieve). after the most similar evidences to a claim are retrieved, they are used to prompt llm together with a claim. from the sentence below, please formulate 1 question that could be answered with this question. this question and answer should help to do the fact checking for the claim that is also given. which question would be asked to get this asnwer given that we need to know whether the claim is true? examples: claim: ... answer: ... question: ... ... based on an llm response, one of the labels, supported, refuted, not enough evidence, conflicting evidence/cherrypicking, is assigned. ## results we report q, q+a, and veracity@0.25 scores in ## conclusion we have described our uhh system that is submitted to the averitec shared task. we have explored the use of rag in this task and have used different llms in different steps, with a different number of evidences -top {3, 5, 7, 10}. top 10 evidences using mixtral-8x7b-instruct-v0.1 (quantized 4bit) model by prompting individual evidence (strategy 2-2) in the retrieve-question approach are ranked 6th in the shared task. in future work, we would like to investigate using a vector database. we have used the evidences as provided by organizers, and we also plan to experiment with different granularity of texts from these evidences.