Evidence Selection as a Token-Level Prediction Task (2021)

Abstract: In Automated Claim Verification, we retrieve evidence from a knowledge base to determine the veracity of a claim. Intuitively, the retrieval of the correct evidence plays a crucial role in this process. Often, evidence selection is tackled as a pairwise sentence classification task, i.e., we train a model to predict for each sentence individually whether it is evidence for a claim. In this work, we fine-tune document level transformers to extract all evidence from a Wikipedia document at once. We show that this approach performs better than a comparable model classifying sentences individually on all relevant evidence selection metrics in FEVER. Our complete pipeline building on this evidence selection procedure produces a new state-of-the-art result on FEVER, a popular claim verification benchmark.

## introduction automated claim verification is the task of determining the veracity of a claim given evidence retrieved from a knowledge base. a popular and large-scale benchmark for this task is fever, consisting of almost 200k synthetically generated claims derived from the introduction sections of wikipedia sentences @xcite . input to such a system is a claim. we then search relevant documents from wikipedia, and select the evidence sentences from the retrieved documents. finally, we determine the veracity of the claim given the retrieved evidence, i.e., we predict one of the labels supports, refutes or not enough info. in this paper, we propose a new angle to think about evidence selection. unlike the sentence-level approaches taken in previous work, we extract all evidence sentences from a wikipedia document at once. this procedure outperforms a comparable sentence-level baseline in fever. the code and models described in this paper are publicly available. 1 consider the claim cary elwes was born in 1982. the fever task consists of extracting the annotated evidence from wikipedia and to predict the right label. for this claim, we would have to retrieve the page about cary elwes from which we have to select the following evidence sentence: cary elwes, born 26 october 1962, is an english actor and writer. because the evidence contradicts the claim, the claim is refuted. fever score, the official metric, is a combination of returning the correct evidence and predicting the correct label. past work considered evidence selection to be a pairwise sentence classification task. for example, the page about cary elwes contains four sentences. a model would predict @xmath0, @xmath1. our intuition is that more context helps to decide whether a given sentence is evidence -where more context is having access to the complete article. hence, we predict a score for each token in a document and aggregate token scores on a sentence level. we show a visualization of our approach in figure 1 . input to our system @xmath2, @xmath3. we then fine-tune a transformer to predict 1 for each token belonging to annotated evidence for a claim, and to predict 0 for all other tokens. at test time, we average scores for all tokens on a sentence level. if the resulting average is higher than 0.5, we consider the sentence to be evidence. because wikipedia articles are often longer than the maximum sequence length allowed in standard transformers (e.g. @xcite , we use bigbird, which uses sparse attention to accommodate long sequences @xcite . this perspective on evidence selection is more related to, e.g., answer span extraction in question answering @xcite . @xcite and @xcite have both shown that long document encoders significantly improve model performance in question answering tasks with lots of context. we explore whether this holds for claim verification in fever as well. for example, consider the claim st. anger is the first studio album by metallica. one evidence sentence for this claim is it was released in june 2003 as the lead single from their eighth studio album of the same name . which can be found in the wikipedia page st. anger (song). without additional context, it is not possible to resolve their eight studio album, and therefore it is difficult to select this sentence as evidence. however, the first sentence of this article states [...] is a song by american heavy metal group metallica ., and thus resolves their. having access to the whole document makes it straightforward to correctly predict this example. to summarize: we propose a new perspective on evidence retrieval in automated claim verification. 2 we treat evidence selection as a token-level prediction task and extract all evidence from a wikipedia article at once. this approach improves all relevant evidence retrieval metrics in fever compared to a roberta baseline. finally, we fine-tune a claim verification model using evidence retrieved by the proposed method which results in a new state-ofthe-art result in fever. ## methods and related work in this section, we describe our methods and we place our modeling decisions in the context of previous fever systems. to begin with, there exists a plethora of related work on fever -the most notable arguably is the fever baseline system itself. @xcite perform (i) coarse-grained tf-idf-based document retrieval, followed by (ii) more fine-grained evidence retrieval and eventually (iii) a model prediction of whether the extracted evidence entails the claim. most subsequent systems broadly follow this pipeline while improving the individual building blocks with more tailormade solutions. and so do we -in this work, we specifically provide a new perspective on evidence selection. @xcite . given the importance of having retrieved the relevant candidate documents, more recent systems started to combine the mediawiki api with tf-idf or bm25 @xcite . ## conclusion we present a novel approach to evidence retrieval. instead of considering sentence selection as a pairwise sentence task, we extract all evidence sentences from a wikipedia article at once by predicting for each token whether it is part of an evidence sentence or not. our method outperforms a comparable roberta checkpoint trained @xmath4, @xmath5, while being less computationally expensive. hence, we consider our method for evidence selection to be an unambiguous improvement over previous approaches. we believe most fever systems (and possibly other claim verification tasks) would benefit from the method proposed in this work. we have shown that our system performs only 1% point worse than an oracle in terms of fever score assuming gold veracity labels. hence, we reason that our evidence selection method reflects strong performance given our initial set of retrieved documents. finally, our full pipeline leads to a new state-of-the-art on the blind fever test set at the time of submission. we consider our work only as the starting point in the incorporation of document-level transformers to automated claim verification. avenues for future work could include e.g. taking a more squadlike approach where only start and end tokens of evidence spans are predicted, or where we predict a special [cls] token for each sentence, while having the whole article as input. given re-ranked documents (such that the combined sequence length @xmath6, it might also be interesting to concatenate all of them and extract evidence at once from all documents for a given claim. or predict the veracity directly from these documents, without having to extract evidence at all.