SK _ DU Team: Cross-Encoder based Evidence Retrieval and Question Generation with Improved Prompt for the AV eri T e C Shared Task (2024)

Abstract: As part of the AVeriTeC shared task, we developed a pipelined system comprising robust and finely tuned models. Our system integrates advanced techniques for evidence retrieval and question generation, leveraging cross-encoders and large language models (LLMs) for optimal performance. With multi-stage processing, the pipeline demonstrates improvements over baseline models, particularly in handling complex claims that require nuanced reasoning by improved evidence extraction, question generation and veracity prediction. Through detailed experiments and ablation studies, we provide insights into the strengths and weaknesses of our approach, highlighting the critical role of evidence sufficiency and context dependency in automated fact-checking systems. Our system secured a competitive rank, 7th on the development and 12th on the test data, in the shared task, underscoring the effectiveness of our methods in addressing the challenges of real-world claim verification.

## introduction fact-checking has become an essential tool in the fight against misinformation, which can have farreaching impacts on public opinion and policy. manual fact-checking is a resource-intensive process, requiring skilled analysts to meticulously scrutinise claims and verify their authenticity. this necessity has driven the development of automated fact-checking (afc) systems designed to assist human fact-checkers by efficiently processing large volumes of information and detecting false claims. @xcite . the effectiveness of afc systems depends significantly on the quality of the datasets used to train and evaluate them. common datasets, such as fever @xcite , feverous @xcite and multifc @xcite , have been instrumental in advancing afc research, but come with limitations, including the reliance on artifi-cially constructed claims and inadequate evidence annotations @xcite . in response to these limitations, the 2024 averitec (automated verification of textual claims) task was specifically designed to address the challenges of real-world claim verification @xcite . averitec comprises 5,783 claims sourced from 50 fact-checking organisations, collected via the google factcheck claim search api. each claim in the dataset is meticulously annotated with question-answer pairs, supported by online evidence, and accompanied by textual justifications explaining how the evidence leads to a verdict. this structured annotation approach ensures that the dataset supports robust afc model training and evaluation @xcite . this advancement aligns the dataset more closely with real-world scenarios, potentially enhancing the generalisation ability of the developed models and facilitating the creation of more robust approaches. the diversity of the data presents unique challenges, necessitating a deeper understanding of the data and the development of effective reasoning strategies. our method (sk_du) achieved the 12 th rank in the averitec shared task during the testing phase foot_0 , providing valuable insights into the strengths and weaknesses of our pipeline and highlighting areas for further improvement. in this paper, we aim to describe the design of our proposed fact verification pipeline and to share the insights we gained on the averitec dataset @xcite during the workshop competition. the paper introduces a comprehensive approach to real-world claim verification, leveraging the averitec dataset to develop and evaluate a sophisticated pipeline for automated fact-checking. the proposed system incorporates cutting-edge models and techniques, including cross-encoders for precise evidence retrieval/reranking @xcite and large language models (llms) for effective question generation @xcite , and cross-encoder based natural language inference (nli) for veracity prediction @xcite . by focusing on multi-stage processing-ranging from the selection of evidence to nuanced reasoning for claim validation, the work addresses the complexities of real-world data, emphasising the importance of context and evidence sufficiency in fact-checking processes. our code is released to the public for further exploration foot_1 . in short, the contributions of this paper are the following: ## dataset insights averitec consists of 5,783 claims sourced from 50 reputable fact-checking organisations, where 4,568 claims' data were released earlier, while 1,215 were released during the testing phase of the averitec shared task foot_2 . each claim is annotated with detailed question-answer (qa) pairs as evidence, a veracity label, and a textual justification, ensuring a robust foundation for training and evaluating afc systems @xcite . additionally, the meta-data information, e.g., speaker, date, url, location, etc., provides contextual details to the claim to support questions, answers, and justifications. this structured and meticulous approach aims to bridge the gap between academic research and practical application in building systems for misinformation detection. as the claims in averitec are also annotated with date, the dataset is split temporally (ordered by date) into training, validation, and test sets, having 500, 3,068, and 2,215 claims data, respectively. reasoning about evidence is structured through a question-and-answer format, allowing for multiple answers to reflect potential disagreements. multihop reasoning is also allowed by referring to previous questions, and all answers must be backed by source urls. in the averitec dataset, the veracity of claims is predicted into typical classes: supported, refuted, and not enough evidence. averitec also introduces a fourth class: conflicting evidence/cherry-picking, which includes conflicting evidence and technically true claims that mislead by omitting crucial context. this addition addresses real-world scenarios where sources may legitimately disagree on interpretations. one of the primary challenges is context dependence. many claims cannot be accurately verified without additional context that is not always available in the fact-checking articles. this lack of context can lead to incorrect or incomplete verification outcomes. another major challenge is evidence sufficiency. ensuring that the evidence provided is comprehensive enough to support or refute claims is crucial, as incomplete evidence can skew the verification results. temporal leakage is another critical challenge, where evidence published af-ter the claim date may inadvertently influence the verification process. this can result in biased or inaccurate conclusions, undermining the integrity of the dataset. additionally, the diverse nature of the data from various sources and the wide range of claim types introduce complexity in data annotation and processing, making it difficult to maintain consistency and accuracy across the dataset. ## results the proposed pipeline's evaluation involved a comprehensive analysis of performance across various stages, including evidence retrieval, evidence selection, and veracity prediction. the results highlight the effectiveness of the proposed approach in handling the complexities of real-world claim verification and the challenges encountered during the process. ## conclusion in this paper, we presented a comprehensive pipeline for real-world claim verification tailored to the averitec dataset. our approach, which integrates cross-encoders for evidence retrieval and llms for question generation, has shown to be effective in improving the accuracy of automated factchecking systems. we show that the cross-encoder performs better than the baseline on both evidence extraction and reranking. the results of our experiments highlight the importance of multi-stage processing and the careful selection of evidence to support or refute claims. our model's performance in the averitec shared task demonstrated its potential in real-world applications, particularly in scenarios requiring detailed reasoning and context understanding. although our system has made sig-nificant strides in addressing the complexities of real-world claim verification, further improvements are necessary, particularly in handling ambiguous claims and ensuring the completeness of evidence.