Verdict Inference with Claim and Retrieved Elements Using R o BERT a (2021)

Abstract: Automatic fact verification has attracted recent research attention as the increasing dissemination of disinformation on social media platforms. The FEVEROUS shared task introduces a benchmark for fact verification, in which a system is challenged to verify the given claim using the extracted evidential elements from Wikipedia documents. In this paper, we propose our 3rd place three-stage system consisting of document retrieval, element retrieval, and verdict inference for the FEVEROUS shared task. By considering the context relevance in the fact extraction and verification task, our system achieves 0.29 FEVEROUS score on the development set and 0.25 FEVEROUS score on the blind test set, both outperforming the FEVEROUS baseline.

## introduction the large-scale dissemination of disinformation on social media platforms intended to mislead or deceive the general population has become a major societal problem @xcite . for example, the widespread disinformation of the covid-19 vaccine has caused a growth of anti-vaccination sentiment online and led to declining vaccination coverage. as the best way to stop disinformation from going viral online is early verification, recent researchers have put efforts into automatic fact verification systems. to answer the increasing demand for such systems, the fever (fact extraction and verification) dataset @xcite was introduced and used for the shared task of the fever workshop 2018. it consists of 185,445 annotated claims with a label of "supported", "refuted", or "not enough info" as well as sets of evidential sentences from the given pre-processed wikipedia pages. among the participated teams of the shared task, @xcite proposed a system consisting of three connected homogeneous networks of document retrieval, sentence selection, and claim verification. @xcite proposed a four-stage system that utilizes logistic regression models for the document retrieval and sentence retrieval stages, enhanced sequential inference model (esim) @xcite for the natural language inference stage, and multi-layer perceptron (mlp) for the aggregation stage. to explore the ability of automatic fact verification systems over both unstructured sentences and structured table-based information, @xcite introduces the fact extraction and verification over unstructured and structured information (feverous) dataset. the shared task in 2021 uses the feverous dataset and further requires a system to be able to retrieve structured information from wikipedia as evidence for each claim, which differs from the shared task in 2018. however, these two shared tasks still share the similar setting as a fact extraction and verification problem, which makes the pipelines and methods of the early proposed systems worth referring to. all in all, the feverous shared task in 2021 challenges a system to extract evidential elements, primarily sentences and table cells, from the given 5.4m wikipedia documents and verify as "supports", "refutes", or "not enough info" for each given claim. systems are evaluated by jointly considering how complete the relevant wikipedia elements are retrieved and how correct the final verification verdicts are. in this paper, we propose a three-stage system as figure 1 shows to improve the feverous baseline in two aspects. first, while the baseline retriever pays attention to literal relevance and word frequency with a combination method of entity matching and tf-idf, we fine-tune the bert model @xcite to integrate the con-text relevance for finding evidential elements and downstream verdict inference. second, the baseline predictor uses the claim and the concatenation of retrieved elements as input, having a constraint of the maximum input length. we experiment with several ways to include more elements for verdict inference. finally, these improvements allow us to achieve substantially higher performance than the baseline. ## system description our system is a three-stage model consisting of document retrieval, element retrieval, and verdict inference. document retrieval aims to extract the selection of the most related wikipedia documents when only given the claim. the claim and the set of candidate elements from the most related wikipedia documents are then given to the subsequent element retrieval to find out the most evidential elements regarding the claim. the final stage utilizes the nli model for verdict inference, predicting the final verdict based on the most evidential elements and the claim. ## error analysis our system proves that performing document retrieval, element retrieval and verdict inference in the three-phase procedure is a proper pipeline for the fact extraction and verification shared task. for evidence retrieval (document and element retrieval), our proposed methods of the bm25-based anserini and the context-aware bert model considers both the presence of certain keywords and semantic context. hence, it is able to extract related elements over unstructured sentences and structured table cells. nevertheless, the retrieval performance still has room for improvement in two aspects. one is to tune the balance weighting between the presence of certain keywords and the semantic context. the other is to attentively design the fine-tuning of the bert model for element retrieval. during the finetuning process, we use the output from the document retrieval stage and set as negative labels for all elements from the 5 most related documents that are not annotated as evidence of the corresponding claim. while each claim has only a maximum of 3 evidence sets and an average of nearly 4 elements per set, our bert model suffers from the unbalanced labeling problem, in which the process includes massive negative and few positive instances. the positive instances in the fine-tuning are also rather few because we use the output of the document retrieval stage that only retrieves a document coverage of 69%. ## conclusion we describe our 3 rd place system for the fever-ous shared task via the three-stage setup of document retrieval, element retrieval, and verdict inference. by considering the context relevance in the fact extraction and verification task, our system achieves 0.29 feverous score on the development set and 0.25 feverous score on the blind test set, both outperforming the feverous baseline.