A Fact Checking and Verification System for FEVEROUS Using a Zero-Shot Learning Approach (2021)

Abstract: In this paper, we propose a novel fact checking and verification system to check claims against Wikipedia content. Our system retrieves relevant Wikipedia pages using Anserini, uses BERT-large-cased question answering model to select correct evidence, and verifies claims using XLNET natural language inference model by comparing it with the evidence. Table cell evidence is obtained through looking for entity-matching cell values and TAPAS table question answering model. The pipeline utilizes zero-shot capabilities of existing models and all the models used in the pipeline requires no additional training. Our system got a FEVEROUS score of 0.06 and a label accuracy of 0.39 in FEVEROUS challenge.

## introduction misinformation on online mediums has caused several problems in recent years. for instance, during the initial spread of the covid-19 pandemic, inappropriate treatments or incorrect statistics have been widely disseminated through posts. manually checking the content of such posts against the fact checking sites is not feasible as it is labor intensive. as a remedy, many automated fact-checking solutions have started to emerge in the last decade. to challenge researchers and advance the domain in this research area, the fact extraction and verification (fever) @xcite challenge was introduced in 2018. this challenge contained 185,445 claims, and the most successful group @xcite obtained a 0.63 fever score in the test set. in 2021, a new challenge, fact extraction, and verification over unstructured and structured information (feverous) @xcite was organized with a new dataset comprising 87,026 claims where the average length of the claims increased significantly. a wikipedia dump with more than 5.4 million articles was provided for claim verification, which included sentences and other page elements such as lists and table cells as potential evidence while the previous challenge's dataset contained only sentences. moreover, the total number of page elements included in the dump increased significantly compared to the previous challenge. although feverous challenge contains less number of claims, it has a higher complexity than fever challenge. in this challenge, participants were not only required to label each claim as "supports," "refutes," or "not enough info" but also provide the correct evidence for it. the baseline model in feverous challenge obtained around 18% feverous score. this model contains two steps, which are retrieval and verdict prediction. the model firstly retrieves relevant pages and then sentences and cells separately from each page. during cell retrieval, tables are linearized to obtain the most relevant cells. then cell retrieval is handled as a binary sequence labeling task. verdict prediction is made using the robustly optimized bert pretraining approach (roberta) model @xcite . in addition, the feverous score assumes that the prediction is correct when the label is correct and a set of evidence is present in the predicted evidence. in this challenge, we developed a pipeline that utilizes zero-shot learning capabilities of existing models where we have considered claims as a question and our retrieved documents as a solution text instead of extracting cells and sentences after the document retrieval. we applied different question answering (qa) models to solve the claim for sentences and cells. we obtained our labels from sentences by using natural language inference (nli) model. after that, we added the cells after the sentence solutions. our model obtained 0.06 fever-ous score, 0.39 label accuracy, and 0.06 evidence f1 score. logistic regression logistic regression esim + aggregation team athene @xcite mediawiki api ## method in the following subsections, we will explain how our system works. an overview of the system in the form of pseudocode is given in algorithm 1. ## results and discussion based on the official leaderboard @xcite , our pipeline's scores along with the baseline, minimum, and maximum feverous scores are shown in ## conclusion in this work, we proposed a fact extraction and verification pipeline that mainly uses anserini to retrieve documents, a bert-based question answering model to retrieve textual evidence, tapas to retrieve table cell evidence, and an xlnet-based entailment model to judge the claim without finetuning them. we believe parameter optimization and challenge-specific fine-tuning can significantly improve the results.