RAG -Fusion Based Information Retrieval for Fact-Checking (2024)

Abstract: Fact-checking involves searching for relevant evidence and determining whether the given claim contains any misinformation. In this paper, we propose a fact verification system based on RAG-Fusion. We use GPT-4o to generate questions from the claim, which helps improve the accuracy of evidence retrieval.Additionally, we adopt GPT-4o for the final judgment module and refine the prompts to enhance the detection accuracy, particularly when the claim contains misinformation. Experiment showed that our system achieved an AVeriTeC Score of 0.3865 on the AVeriTeC test data, significantly surpassing the baseline score of 0.11.

## introduction in recent years, misinformation has become easier to spread online @xcite . consequently, to prevent its spread, the demand for automated fact-checking, which automatically detects unreliable information has significantly increased @xcite . fact-checking involves searching for information necessary for verification (evidence) from reliable external databases, and determining the truthfulness of given claim based on that information @xcite . there are various fact-checking datasets, with unstructured data like text @xcite and structured data like tables @xcite or knowledge graphs @xcite . generally, these datasets include a claim, the evidence that needs to be searched to verify the claim, and a label indicating the judgment. for example, in fever @xcite , claims need to be classified into three labels: "supported", "refuted", or "not enough information". numerous systems have been proposed @xcite @xcite , and the accuracy of this three-class clas-sification has reached nearly 0.8 foot_0 . however, the claims included in these datasets are created from sources like wikipedia for specific purposes, and they differ from the claims that journalists actually verify. there is a dataset that include real-world data @xcite , but they face the issue of not providing sufficient evidence necessary for judgment @xcite . in this shared task, averitec @xcite has been newly created. in averitec, the evidence is based on information collected from the web and is provided in a question-answer pair format by human annotators. the judgment labels are: "supported", "refuted", "not enough evidence (nee)", and "conflicting evidence/cherrypicking". additionally, for each claim, the reasons why annotators assign the judgment labels are annotated. the system needs to extract evidence from documents obtained through web searches or from documents provided by the organizers as web search results, and then predicts the claim label. the claim is considered correctly judged only if the necessary evidence is appropriately retrieved, and the final judgment label is correctly predicted. in this paper, we designed the system shown in figure 1 to improve the averitec baseline. the baseline system primarily used bm25 @xcite for evidence collection, but this method does not allow for searching based on the meaning of the claim or web document. therefore, we perform searches using embedding vectors with stella_en_400m_v5 foot_1 . we generate embedding vectors for the claim and the document, and collect 50 documents related to the claim based on their similarity. next, inspired by rag-fusion @xcite , we use gpt-4o to generate three questions from the claim that are needed to search for the evidence. for each of these generated questions, we select three answer sentences from the previously collected 50 documents. these question-answer pairs collected through this procedure are input into gpt-4o along with the claim for the final judgment in verdict inference. the proposed fact-checking system achieved an averitec score of 0.3865 on the test data. ## system description the system we developed is structured in three phases similar to @xcite : document retrieval, question generation and sentence retrieval and verdict inference. document retrieval: since the document set provided by the organizers is vast, this phase selects documents related to the claim. question generation and sentence retrieval: referring to the rag-fusion method, questions for information retrieval are generated using gpt-4o from the claim. subsequently, the sentences that answer these generated questions are retrieved from the sentences contained within the documents selected in the document retrieval phase. verdict inference: using gpt-4o, which has high inferential capabilities, a judgment is made based on the obtained question-answer pairs and the claim. we use gpt-4o via openai api foot_2 . ## result in this chapter, we explain the results at each phase of the system. to consider improving search accuracy, we report the experimental results using a validation dataset (containing 500 claims) where the correct evidence has been distributed. additionally, when using gpt-4o, the temperature is set to 0 to ensure the reproducibility of the experiments. ## error analysis figure 4 (a)(b) shows the confusion matrix when the correct data or retrieved data using a rag-fusion-based search is provided. it can be seen that when the correct label is "nee (not enough evidence)" or "conflict", there is a tendency to predict it as "refuted". this is likely due to the instruction included in the prompt: "if there is even the slightest possibility that it is incorrect, output 'refuted'." however, in fact-checking, to accurately predict "refuted" claims as refuted is the most important. since it is crucial not to provide the user with incorrect information, it is undesirable to remove this instruction from the prompt. figure 4 (c) shows the confusion matrix when this instruction is removed and the correct evidence is provided, revealing an increased risk of failing to detect refuted claims, even when the information is complete. to address this, adopting the concept of corrective retrieval augmented generation (crag) @xcite could be considered for "nee". in crag, a new module is introduced to determine whether the retrieved document is necessary or not. if we incorporate the module into our system, we could first determine whether the information is enough or not. if the information is not enough, the system would classify it as "nee". if the information is enough, the system would proceed to classify the remaining three classes using the similar prompt as in 2(d). by adopting this new module, we will be able to improve the performance of "nee". as a test, using gpt-4o, we performed a twoclass classification-whether the information was complete-using the prompt from figure 2(e ) with the correct data provided. in this task, "supported", "refuted", and "conflict" were considered as having complete information, while "nee" was considered as lacking information. the accuracy rates were 90% for "supported", 86% for "refuted", 60% for "nee", and 78% for "conflict". therefore, further prompt improvements are needed to adapt gpt-4o to this two-stage approach. finetuning bert should also be considered. the "conflict" class is difficult to render a verdict on, so further improvements will be necessary. ## another approach in this section, we will introduce a classification approach that we experimented with but did not yield satisfactory results. although the performance did not exceed that of gpt-4o's 4-class classification, we will present it here in the hope that it may contribute to future efforts by other participants. we considered fine-tuning bert as the final classifier for 4-class classification. however, the dataset exhibits a bias in the classification labels (in the training data: "supported" 27.6%, "refuted" 56.8%, "not enough evidence (nee)" 6.4%, "conflicting evidence/cherry-picking" 9.2%). in particular, the "nee" and "conflict" labels are underrepresented. to address this, we devised two separate classifiers: one for "supported" and another for "refuted". these classifiers perform binary classification, with the supported classifier determining whether a claim is "supported" or not, and the "refuted" classifier determining whether a claim is "refuted" or not. the final prediction label for the claim is then determined based on the results of these classifiers. if the supported classifier predicts true and the refuted classifier predicts false, the final prediction is "supported". conversely, if the supported classifier predicts false and the refuted classifier predicts true, the final decision is "refuted". if both classifiers predict false, the decision is "nee", and if both predict true, it is "conflict". this approach can mitigate the issue of label imbalance. for example, in the supported classifier, claims that are annotated as "supported" are used as positive examples, while "refuted" and "nee" claims are used as negative examples. this allows for similar treatment of "refuted" and "nee" labels. we fine-tuned bert-base-uncased foot_4 for both a 4class classifier and the combined two-classifier approach (batch @xmath0, learning @xmath1, with the training data split 9:1 and used for fine-tuning). the label accuracy on the validation data, when provided with correct evidence, was 0.536 for the 4-class classifier and 0.60 for the combined twoclassifier approach. these results indicate that combining the two classifiers yields higher accuracy. however, as shown in the fourth row of ## conclusion this paper discusses a method for solving the averitec task. the proposed system, inspired by rag fusion, pre-generates questions for information retrieval. this approach allows for a greater amount of information to be used in searches compared to using only the claims. the label accuracy and averitec score showed that pre-generating questions resulted in higher accuracy. proposing an evaluation metric that can consider information beyond the currently accepted evidence when making judgments may lead to more appropriate progress in future research and development. given the rapid advancement of llms, there is also a need to conduct research on adopting llms for the evaluation of evidence validity.