Understanding the Impact of Evidence-Aware Sentence Selection for Fact Checking (2021)

Abstract: Fact Extraction and VERification (FEVER) is a recently introduced task that consists of the following subtasks (i) document retrieval, (ii) sentence retrieval, and (iii) claim verification. In this work, we focus on the subtask of sentence retrieval. Specifically, we propose an evidence-aware transformer-based model that outperforms all other models in terms of FEVER score by using a subset of training instances. In addition, we conduct a large experimental study to get a better understanding of the problem, while we summarize our findings by presenting future research challenges.

## introduction recently a lot of research in the nlp community has been focused on the problem of automated fact checking @xcite . in this work, we focus on the fever dataset that is the largest fact checking dataset @xcite . the goal of the task is to identify the veracity of a given claim based on wikipedia documents. the problem is traditionally approached as a series of three subtasks, namely (i) document retrieval (select the most relevant documents to the claim), (ii) sentence retrieval (select the most relevant sentences to the claim from the retrieved documents), and (iii) claim verification (validate the veracity of the claim based on the relevant sentences). several models have been proposed for the fever dataset @xcite @xcite . most of the existing literature @xcite focuses on the task of claim verification, while little work has been done on the tasks of document retrieval and sentence retrieval. we suspect that this is because it is more straightforward for researchers to focus only on the improvement in terms of performance of the last component (i.e., claim verification) instead of experimenting with the whole pipeline of the three subtasks. in addition, the performance in the first two components is already quite high (i.e., >90% in terms of document accuracy for the document retrieval step @xmath0. unlike the aforementioned studies, in this work, we focus on the task of sentence retrieval on the fever dataset. specifically, inspired by studies that investigate the impact of loss functions and sampling on other domains (e.g., computer vision @xcite , information retrieval @xcite ), this paper -to the best of our knowledge -is the first attempt to shed some light on the sentence retrieval task by performing the largest experimental study to date and investigating the performance of a model that is able to take into account the relations between all potential evidences in a given list of evidences. the contributions of our work are as follows: (i) we propose a simple yet effective evidence-aware transformer-based model that is able to outperform all other models in terms of the fever score (i.e., metric of the claim verification subtask) and improve a baseline model by 0.7% even by using a small subset of training instances; (ii) we conduct an extensive experimental study on various settings (i.e., loss functions, sampling instances) showcasing the effect in performance of each architectural choice on the sentence retrieval and the claim verification subtasks; (iii) the results of our study point researchers to certain directions in order to improve the overall performance of the task. sentence and the potential evidence sentence is a "premise" sentence. in fig. 1 , we present the various architectures that we used in our experiments. ## related work an extensive review on the task of fact extraction and verification can be found in @xcite . for the sentence retrieval task, several pipeline methods @xcite ) rely on the sentence retrieval component of @xcite that use tf-idf representations. an important line of research @xcite @xcite includes the use of esimbased models @xcite . those works formulate the sentence selection subtask as an nli problem where the claim is the "premise" sentence and the potential evidence sentence is a "hypothesis" sentence. similar to the esim-based methods, language model based methods @xcite @xcite @xcite transform the sentence retrieval task to an nli problem using pre-trained language models. for the language model based sentence retrieval two types of losses have been exploited (i) pointwise loss, and (ii) pairwise loss, as presented also in section 2. unlike the aforementioned studies that rely only on losses of type (i) and (ii), we conduct the largest experimental study to date by using various functions on the sentence retrieval subtask of the fever task. in addition, we propose a new evidence-aware model that is able to outperform all other methods using a limited number of training instances. ## conclusion in this paper, we focus on the subtask of sentence retrieval of the fever task. in particular, we propose a simple and effective evidence-aware model that outperforms all other models in which each potential evidence takes into account information about other potential evidences. the model uses only a few training instances and improves a simple pointwise loss by 0.7% percentage points in terms of fever score. in addition, we conduct a large experimental study, compare the pros and cons of the studied architectures and discuss the results in a comprehensive way, while pointing researchers to future research directions.