FZI - WIM at AV eri T e C Shared Task: Real-World Fact-Checking with Question Answering (2024)

Abstract: This paper describes the FZI-WIM system at the AVeriTeC shared Task, which aims to assess evidence-based automated fact-checking systems for real-world claims with evidence retrieved from the web. The FZI-WIM system utilizes open-source models to build a reliable fact-checking pipeline via question-answering. With different experimental setups, we show that more questions lead to higher scores in the shared task. Both in question generation and question-answering stages, sampling can be a way to improve the performance of our system. We further analyze the limitations of current open-source models for real-world claim verification. Our code is publicly available https://github.com/jens5588/FZI-WIM-AVERITEC.

## introduction disinformation is a major concern in digital times as recent advances in generative artificial intelligence, i.e., large language models (llms), enable humans to create fake information on a large scale. meanwhile, llms have also been integrated into automated fact-checking (afc) systems @xcite , which have drawn lots of attention. @xcite summarize three stages of an afc system: claim detection, evidence retrieval, and claim verification. various evidence-based fact-checking datasets have been proposed for testing the systems @xcite @xcite . the averitec shared task aims to fact-check real-world claims. compared to previous fact-checking datasets, the averitec dataset @xcite utilizes questionanswer (qa) pairs to tackle the complex reasoning task for real-world claims. questioning is a natural step in the fact-checking process. the following steps involve retrieving corresponding answers and making inferences based on the qa pairs to validate the claims. @xcite have introduced the qabrief dataset, which was collected via crowdsourcing. they demonstrate that generating questions and then answering questions using open-domain question-answering can increase the accuracy and efficiency of fact-checking. with the claimdecomp dataset, @xcite show that questions to the claim can help identify relevant evidence and verify the claim with their answers. the fzi-wim system is composed of three stages, namely, question generation, questionanswering, and claim verification. all components in the system are designed with open-source models. given the claim and its meta information, the system first generates critical questions. a retrieval augmented generation (rag) system is utilized to answer the generated questions with context information from the provided knowledge store. the generated qa pairs are fact-checked and filtered to tackle the potential hallucination problem. the selected qa pairs are utilized to verify the claim. we summarize our findings regarding this shared task as follows: ## background the averitec dataset @xcite is a continuation of the previous evidence-based fact-checking dataset fever @xcite and feverous @xcite . the dataset contains real-world claims from various sources. the number of claims in the train, dev, and test set are 3068, 500, and 2215 respectively. there are five types of claims in the dataset, namely position statement, numerical claim, event/property claim, quote verification, and causal claim. the corresponding evidence has been collected from internet websites. different from the previous dataset, which uses sentences from documents as evidence, the evidence of the averitec dataset has been formulated as qa pairs. on average, each claim in the train and dev sets has 2.6 questions. the answers can be classified into four types, boolean, abstractive, extractive, and unanswerable. based on the qa pairs, the verification labels of the claims can be classified into supported, refuted, not enough evidence, and conflicting evidence/cherry-picking. figure 1 shows an example from the dataset. claim: donald trump has kept his promises to voters. claim type: event/property claim speaker: none claim date: 24-8-2020 question 1: what promises did donald trump make to voters? answer 1 (extractive & abstractive): during the 2016 campaign, donald trump made more than 280 promises, though many were contradictory or just uttered in a single campaign event. by 2020 trump had made a number of promises, 6 of which he had not fulfilled, including ... question 2: of the promises donald trump made, did he fulfil any of them? answer 2 (boolean): yes. question 3: has president donald trump kept his campaign promises to voters? answer 3 (abstractive): president trump has only kept a few of his promises. verification: conflicting evidence/cherrypicking justification: qa pairs state promises kept and not kept. claim does not state he kept all promises. 3 system description figure 2 illustrates the three-stage pipeline of the fzi-wim system for the averitec shared task in the test phase. in the following, we will describe the key components of each stage. the technical implementation details are presented in appendix a.1. ## evaluation in this section, we show the performance of our proposed systems for the shared task. besides the system in the test phase, the fzi-wim test, we also include the improved version in the after competition phase, fzi-wim after compet., for comparison. with the fzi-wim after compet. setup, each claim has three sets of distinct questions without conditional check described in section 3.2.3. ## conclusion & outlook in this paper, we have described the fzi-wim system for the averitec shared task, which aims to tackle the real-world claim verification problem. the complex reasoning problem in fact-checking is tackled via question-answering. for each claim, we first generate relevant critical questions. based on the provided knowledge store, the questions are answered with an rag system. considering the hallucination problem in rag systems, we factcheck the generated qa pairs to ensure the answers are entailed in the source texts. we show that more questions, i.e., more question-answering rounds, lead to better model performance. the claim verification is based on the selected qa pairs. generally, our current systems need a large amount of computing. the improvement of the efficiency with open-source models is needed for the real-world scenario. compared to proprietary models, our generator in the rag system is not robust enough against adversarial contexts, e.g., misinformation, satire, etc. further enhancement of the robustness can be a promising research direction.