AIC CTU system at AV eri T e C : Re-framing automated fact-checking as a simple RAG task (2024)

Abstract: This paper describes our 3^{rd} place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models.We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation.We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative.We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.

## introduction we release a pipeline for fact-checking claims using evidence retrieved from the web consisting of two modules -a retriever, which picks the most relevant sources among the available knowledge store foot_1 and an evidence & label generator which generates evidence for the claim using these sources, as well as its veracity label. our pipeline is a variant of the popular retrievalaugmented generation (rag) scheme @xcite , making it easy to re-implement using established frameworks such as langchain, haystack, or our attached python codebase for future research or to use as a baseline. 2 related work it proposes the averitec score -a method of unsupervised scoring of fact-checking pipeline against this gold data using hungarian meteor score, matching the evidence questions (q) or the whole evidence (q+a). the score is then calculated as the proportion of claims with accurate label and sound ev-idence (using a threshold for hu-meteor such as 0.25) among all claims in the dataset, giving an estimate on "how often the whole fact-checking pipeline succeeds end to end". the provided baseline is a pipeline of search query generation, api search (producing a knowledge store), sentence retrieval, question-and-answer (qa) generation, qa reranking, qa-wise claim classification and label aggregation, achieving an overall averitec test set score of 11%. ## system description our system design prioritizes simplicity, and its core idea is using a straightforward rag pipeline without engineering extra steps, customizing only the retrieval step and llm prompting (listing 1 in appendix a). despite that, this section describes and justifies our decisions taken at each step, our additions to the naive version of rag modules to tune them for the specific task of fact-checking, and their impact on the system performance. ## other examined approaches in this section, we also describe a third, optional module we call the veracity classifier, which takes the claim and its evidence generated by our evidence & label generator (section 3.2) and predicts the veracity label independently, based on the suggested evidence, using a fine-tuned nli model. we also describe the options of its ensembling with veracity labels predicted in the generative step (section 3.2.5). the absence of a dedicated veracity classifier has not been shown to decrease the performance of our pipeline significantly (as shown, e.g., in tables 2 and 1) so we suggest to omit this step altogether and we proceed to participate in the averitec shared task without it, proposing a clean and simple rag pipeline without the extra step (figure 1 ) for the fact-checking task. ## results and analysis we examine our pipeline results using two sets of metrics -firstly, we measure the prediction accuracy and f 1 over predict labels without any ablation, that is obtaining predicted labels using the predicted evidence generated on top the predicted retrieval results. while the retrieval module is fixed throughout the experiment (a full scheme described in section 3.1), various evidence & label generators and classifiers are compared in in real world, however, the evidence quality is critical for the fact-checking task. we therefore proceed to estimate it using the hu-meteor evidence question score, qa score and averitec score benchmarks briefly explained in section 2 and in greater detail in @xcite . we use the provided averitec scoring script to calculate the values for the latter experiments shown in posterior to the original experiments and to the averitec submission deadline, we also compute the pipeline results using an open-source modelthe llama 3.1 70b 13 @xcite obtaining encouraging scores, signifying our pipeline being adaptable to work well without the need to use a blackboxed proprietary llm. ## conclusion in this paper, we describe the use and development of a rag pipeline over real world claims and data scraped from the web for the averitec shared task. its main advantage are its simplicity, consisting of just two decoupled modules -retriever and an evidence & label generator -and leveraging the trainable parameters of a llm rather than on complex pipeline engineering. the llms capabilities may further improve in future, making the upgrades of our system trivial. in section 3, we describe the process of adding features to both modules well in an iterative fashion, describing real problems we have encountered and the justifications of their solution, hoping to share our experience on how to make such systems robust and well-performing. we publish our failed approaches in section 4 and the metrics we observed to benchmark our systems in section 5. we release our python codebase to facilitate further research and applications of our system, either as a baseline for future research, or for experimenting alongside human fact-checkers.