BEVERS : A General, Simple, and Performant Framework for Automatic Fact Verification (2023)

Abstract: Automatic fact verification has become an increasingly popular topic in recent years and among datasets the Fact Extraction and VERification (FEVER) dataset is one of the most popular. In this work we present BEVERS, a tuned baseline system for the FEVER dataset. Our pipeline uses standard approaches for document retrieval, sentence selection, and final claim classification, however, we spend considerable effort ensuring optimal performance for each component. The results are that BEVERS achieves the highest FEVER score and label accuracy among all systems, published or unpublished. We also apply this pipeline to another fact verification dataset, Scifact, and achieve the highest label accuracy among all systems on that dataset as well. We also make our full code available.

## introduction the danger of misinformation online has gained significant attention in recent years. this has been reignited by the recent covid-19 pandemic, where social media sites and other entities were tasked with identifying misleading content or false content to warn users. being able to develop systems to automate or build tools to improve this process could reduce the need for human annotators to mark content as being misleading or false. the fact extraction and verification (fever) dataset @xcite is one the largest and most popular datasets aimed at automated fact verification. the fever dataset is comprised of 185,445 claims and uses a 2017 dump of wikipedia as the corpus to verify the claims, which results in a corpus size of over 5,000,000 articles. for each claim, the task is to find the relevant wikipedia page(s), the relevant sentence(s) within the page(s), and finally given the relevant sentences and claim determine if the claim is supported, refuted, or there is not enough information. as such, a fairly standard pipeline of a document retrieval system, a sentence selection system, and a final claim classification system is used by most of the systems for the task. the primary metric for the dataset is fever score. the fever score requires both that the predicted label is correct as well as at least one piece of correct evidence being retrieved as predicted evidence. much of the recent work has examined parts of the pipeline and made novel improvements over baseline approaches. for our system, rather than making novel improvements against the baseline pipeline, we instead tune each of these components to ensure maximum performance. in fact, our pipeline is quite similar to one of the first fever systems to utilize transformer models @xcite . we call our system baseline fact extraction and verification system (bevers). despite its relative simplicity, our system attains state of the art (sota) performance on the fever blind test set. when applying our baseline pipeline to another popular fact verification dataset, scifact @xcite , our system achieves the highest label f1 score on that dataset as well. ## experimental setup what we believe to be the source of improvements for our system is hyperparameter tuning for each component. we identify hyperparameters and potential values and run a grid search to find the optimal configurations for each component. in this section, we will go over each of the grid searches our sentence selection hyperparameter tuning is split into two sections. first, we optimize the number of negative samples selected as well as binary vs ternary classes for ranking. since the fever dataset does not provide evidence for not enough info claims, negative samples must be used to generate training examples for these. using the best selection from the initial setup, we tune the learning rate and label smoothing. the candidate values for the tuning can be found in the claim classification tuning setup is quite similar to sentence selection. we initially tune the learning rate and label smoothing using the same candidate values for the concatenated case. instead of tuning the model types of singleton, concatenated, and mixed, we simply use the best hyperparameter configuration and train a model for each of these to draw final comparisons. again, given the imbalance in classes in the train set, we use class weighting to compensate for this imbalance. for fine-tuning we use roberta large mnli and deberta v2 xl mnli. ## conclusion we presented bevers, a strong baseline approach for the fever and scifact datasets. despite being similar to previous works in structure @xcite and utilizing little in terms of novel improvements, our system was able to achieve sota performance on fever and the highest label accuracy on scifact. we primarily attribute these improvements to diligent hyperparameter tuning and error analysis. while several previous works have shown novel contributions to portions of the pipeline can yield improvements, in this work we show a well-tuned baseline is very strong.