Retrieving Semantics for Fact-Checking: A Comparative Approach using CQ (Claim to Question) & AQ (Answer to Question) (2024)

Abstract: Fact-checking using evidences is the preferred way to tackle the issue of misinformation in the society. The democratization of information through social media has accelerated the spread of information, allowing misinformation to reach and influence a vast audience. The significant impact of these falsehoods on society and public opinion underscores the need for automated approaches to identify and combat this phenomenon.This paper is describes the participation of team IKR3-UNIMIB in AVeriTeC (Automated Verification of Textual Claims) 2024 shared task. We proposed a methods to retrieve evidence in the question and answer format and predict the veracity of a claim. As part of the AVeriTeC shared task, our method combines similarity-based ColBERT re-ranker with traditional keyword search using BM25. Additionally, a recent promising approach, Chain of RAG (CoRAG) is introduced to generate question and answer pairs (QAs) to evaluate performance on this specific dataset. We explore whether generating questions from claims or answers produces more effective QA pairs for veracity prediction. Additionally, we try to generate questions from the claim rather than from evidence (opposite the AVeriTeC dataset paper) to generate effective QA pairs for veracity prediction. Our method achieved an AVeriTeC Score of 0.18 (more than baseline) on the test dataset, demonstrating its potential in automated fact-checking.

## introduction the volume of potentially misleading and false claims has surged with the increasing usage of the web and social media. a study from 2018 found that false information spreads six times faster than the truth on platforms like twitter @xcite . considering the democratization of information through social media and the need to limit the spread of misinformation, different approaches and dataset have been explored to automate the fact-checking process @xcite . over the years, various datasets have been released to facilitate different solutions by the research community. in this context, this paper presents the various approaches for the averitec (automated verification of textual claims) shared task foot_0 . the averitec @xcite dataset contains 5,782 real-world claims (3,068 for training, 500 for development, and 2,214 for testing) sourced from 50 different fact-checking organizations. each claim is accompanied by annotated question-answer pairs, backed by online evidence, along with textual justifications that explain how the evidence is integrated to produce a verdict. the claims in averitec are classified into four labels: "supported", "refuted", "not enough evidence", and "conflicting evidence/cherry-picking.". the evaluation is performed using the averitec score, that considers the correct retrieval of evidence and credits veracity predictions only when the correct evidence has been found. the primary goal of the proposed system is to enhance the pipeline by offering pertinent evidence. in this paper, we propose different approaches that focus on the retrieval component through a re-ranking process of the retrieved sentences, aiming to achieve increased precision. furthermore, unlike the baseline where the question is generated starting from the sentence, we explore whether generating the question from the claim can improve performance. this idea is combined with the methodology proposed in @xcite , which utilizes the chain of rag (corag), to determine the appropriateness of this technique in this particular case. automatic fact-checking has become a fundamental challenge for research. our paper focuses on analyzing and proposing advanced solutions for fake news detection by enhancing the pipeline with hybrid-search techniques and the use of chain of rag (corag). ## related work the early fact checking datasets, such as fever @xcite , vitaminc @xcite , and feverous @xcite are assembled from wikipedia by corrupting the wikipedia page statements. gradually, the subsequent datasets like liar-plus @xcite ,politihop @xcite , multifc @xcite are developed from the fact checking website. however, these datasets did not address the issues such as context dependence, evidence insufficiency, and temporal leaks. the averitec dataset @xcite proposed in the shared task has addressed all the above issues during the data assembly. the baseline of averitec dataset @xcite consists of several components: (i) search: using google api. (ii) evidence retrieval: initially, bm25 scores are computed against the claim, and sentences outside the top 100 are discarded. (iii) generation of questions given retrieved evidence: for each of the retrieved sentences, a question is generated that is answerable by that sentence, using bloom with 10 in-context examples in question and answer format from the trainset. (iv) re-ranking of retrieved evidence: the retrieved evidence in question-answer pair format is re-ranked to identify the top three most relevant pairs for the claim, using a fine-tuned bert-large model. (v) veracity prediction: the veracity of the claim is predicted using a fine-tuned bert-large model with 340 million parameters. (v) generation of justifications: justifications are generated using a finetuned bart-large model with 406 million parameters. different approaches have been used in recent years in the field of fact-checking. the questionanswer decomposition is considered a promising strategy, as cited in @xcite . a recent work, ragar @xcite , applies this strategy specifically to political fact-checking, proposing two novel methodologies: chain of rag (corag) and tree of @xcite . this approach appears to improve veracity prediction and the generation of justifications compared to traditional fact-checking. corag employs a sequential question strategy, generating questions and related answers as needed for predicting veracity, using follow-up checks. as evidenced in @xcite , these methods have been evaluated as effective and show potential for future research in combating misinformation, suggesting evaluation in other misinformation domains. @xcite proposed a framework in which evidences are grouped into two groups with opposite polarity with respect to the claim. they used llm to generate the justification and the inference module for the veracity prediction to improve the evidence retrieval process, col-bert @xcite is considered due to its effectiveness and efficiency. colbert is a novel ranking model that adapts deep language models (specifically, bert) for efficient retrieval with a fine-grained similarity in respect to the provided query. all the cited works are integrated to develop a solution that achieves high performance in retrieving relevant evidence for accurate veracity prediction. ## system description the proposed architecture for the automated factchecking process can be divided into five parts: ## experiments the rationale behind the suggested solution is to apply a hybrid search approach combining bm25, as carried out in the baseline, and colbert (khattab and zaharia, 2020). this preserves the ease of use and effectiveness of keyword-based matching while enabling to extract semantic meaning at a higher level of detail. the next sections introduce the various configurations that are set up to perform the comparative evaluations. ## results all the experiments introduced and discussed in the previous section are evaluated based on the averitec metrics, available for both test and development sets. averitec score measures how closely the generated question and answer pairs match the gold standard, using hungarian meteor (banerjee and lavie, 2005). in run 1, the performance is better than the baseline, suggesting that the use of semantic search improves the pipeline's effectiveness, as shown in in terms of the test set, the performance on the averitec score is higher for run2-aq (answer to question), indicating that generating questions starting from the answers is more effective, yielding comparable q and q+a scores. in the development set, the f 1 score for each label is available, allowing us to see that the performance is comparable across experiments. in all three runs, the refuted class has an f 1 score around 0.70, indicating that most of the refuted claims are classified correctly, as shown in ## analysis & discussion our improvements to the baseline can be summarized as follows: we enhanced sentence retrieval by focusing on both keyword-based matching and semantic similarity to the claim. this approach allows to capture both keyword-based relevance and semantic alignment, thus retrieving sentences that comprehensively cover various aspects of a claim. this "hybrid search" methodology significantly improves upon the baseline. additionally, we explored and compared two approaches for question generation: generating questions from answers versus generating them from claims. both methods proved effective, but further investigation is needed to determine which is superior. the results indicate that the performance depends significantly on the set of considered claims. the claim-to-question strategy perform better on the development set but unexpectedly underperforms to answer-to-questions on the test dataset. in future, we will try to investigate this inconsistency. a manual check on some claims confirms this: in some cases, the generated questions are more effective in run2-aq, where questions are generated starting from the answer, while in other cases, generating the question from the claim, as done in run3-qa, is more effective. a significant example is the second claim of the test set is the following: "meatpacking workers have suffered more covid-19 cases than healthcare workers." in this context, particularly in the e3-cqa experiment, a very useful question to verify the claim is generated: "what is the rate of covid-19 infection among meatpacking workers in relation to the size of their total workforce compared to the same rate among healthcare workers?". in all other run, for the same claim, questions are generated that consider only healthcare workers or meatpacking workers without comparing the two worker categories together. in other cases, the aq approach-generating questions from answers-is more effective; it depends on the provided claim. generating questions from claims using the chain of rag (corag) approach appears to be more practical for real-world applications. starting from the provided claim, only the questions necessary to verify the claim's veracity are generated, and a corresponding answer is retrieved. in a real-world scenario with a pre-built index, the generated questions are used as queries to retrieve the most suitable answers. this approach reduces the effort required to generate question-answer pairs, thereby shortening the time needed to label claims as 'supported' or 'refuted.' ## future work the averitec dataset contains meta information claim types. we believe that we cannot have the same kind of fact-checking strategy for different types of claims, such as quote verification and comparing numerical quantity. further investigation is needed to determine whether generating questions from answers or from claims is the more effective approach for the claim type. additionally, expanding the colbert component to include more sentences for vectorization could improve the search's ability to find semantically similar sentences. this is discussed in detail in the limitations section. 8. ## conclusion this paper describes various approaches for the fact verification in the averitec shared task @xcite . the method consists of three main parts: the retrieval component, the question generation component, and the veracity prediction component. in the retrieval phase, sentences are initially retrieved using bm25 and then re-ranked based on their semantic similarity to the claim using colbert. for question generation, the previously retrieved sentences are used as answers, with related questions generated through gpt-3.5 in the aq (answer to question) approach. in the second approach, cq (claim to question), the search process is similar, but questions are generated starting from the claim itself. this approach utilizes a recent method called chain of rag (corag) to generate question-answer pairs more effectively. both approaches exhibit effectiveness, with the second approach showing particular promise for real-world applications due to its efficiency in generating questions from claims and building only the necessary q&a pairs for veracity prediction. further advancements are needed, including extending the neural re-ranking phase to more sentences to provide the model with a larger pool of potential answers to the proposed questions. our work contributes to the field by demonstrating the viability of using semantic search and chain of rag techniques in fact-checking and suggests further avenues for research in the evidence retrieval task.