Exploring Listwise Evidence Reasoning with T5 for Fact Verification (2021)

Abstract: This work explores a framework for fact verification that leverages pretrained sequence-to-sequence transformer models for sentence selection and label prediction, two key sub-tasks in fact verification. Most notably, improving on previous pointwise aggregation approaches for label prediction, we take advantage of T5 using a listwise approach coupled with data augmentation. With this enhancement, we observe that our label prediction stage is more robust to noise and capable of verifying complex claims by jointly reasoning over multiple pieces of evidence. Experimental results on the FEVER task show that our system attains a FEVER score of 75.87% on the blind test set. This puts our approach atop the competitive FEVER leaderboard at the time of our work, scoring higher than the second place submission by almost two points in label accuracy and over one point in FEVER score.

## introduction in recent years, the internet has become an effective platform for creating and sharing content to large audiences. unfortunately, there have been occurrences of bad actors taking advantage of this to propagate manipulative information for their benefit, often to the point of spreading misinformation. with the large amount of data being generated on the internet each day, it is infeasible to manually verify it all, motivating recent research into automated fact verification. in this work, we explore a fact verification framework built with the pretrained sequence-tosequence transformer t5 @xcite as its backbone which we call list5. within a standard three-stage architecture, we focus mostly on the label prediction problem. we adopt a "listwise approach", where all candidate sentences that form the evidence set of a claim are considered together. our main contribution is a data augmentation technique that involves deliberately introducing noise into training data to combat data sparsity and produce a more robust model. at its introduction, a full pipeline using our techniques represents the state of the art, achieving the top scoring run on the fever leaderboard. an additional minor contribution exploits named entities during the sentence selection phase, which has a small but noticeable effect on generating a better candidate set for downstream label prediction. we believe that these techniques can be potentially valuable to a broader range of nlp tasks that also involve aggregation of information from upstream retrieval models. ## background and related work as this work focuses on the fact extraction and verification (fever) task @xcite ), foot_0 we begin by briefly describing the task setup. we are given a textual claim q, to be verified against a corpus comprised of a subset of wikipedia. each claim is associated with a three-way veracity label v(q) ∈ {supports, noinfo, refutes} and a set of reference sentences s(q) that provide support. 2 an example claim q, its label v(q), and supporting evidence s(q) are given in figure 1 . the primary evaluation metric, fever score, is computed as the proportion of claims where the system has predicted the correct veracity label conditioned on also having retrieved a complete set of reference sentences. most current systems adopt a three-stage approach to this task, comprising document retrieval, sentence selection, and label prediction. in this work, our contributions are focused on the second and third sub-tasks; for document retrieval, we simply augment current best practices with bm25 @xcite . claim: the rodney king riots took place in the most populous county in the usa. evidence 1 (wiki/los angeles riots): the 1992 los angeles riots, also known as the rodney king riots were a series of riots, lootings, arsons, and civil disturbances that occurred in los angeles county, california in april and may 1992. evidence 2 (wiki/los angeles county): los angeles county, officially the county of los angeles, is the most populous county in the usa. label: supports figure 1 : an example claim and its corresponding evidence and label from the fever dataset. by construction, the veracity of each claim is determined by the (candidate) supporting sentences, taken together. one simple and popular approach to fact extraction and verification is to consider the veracity of the claim with respect to each candidate independently (i.e., classification), and then aggregate the evidence @xcite @xcite @xcite . for convenience, we refer to these as "pointwise approaches", borrowing from the learning to rank literature @xcite . as an alternative, researchers have proposed approaches that consider multiple candidates at once to jointly arrive at a veracity prediction @xcite @xcite @xcite . for convenience, we refer to these as "listwise approaches", also borrowing from the learning to rank literature @xcite . such listwise approaches have also been used for information aggregation in other nlp tasks such as question answering @xcite . at a high level, this strategy suffers from a number of challenges, including data sparsity and a high level of sensitivity to noisy inputs. following this thread of work, we adopt the listwise approach and improve it by training with a data augmentation technique that involves deliberately introducing noise into the training data to produce a more robust model. ## methods our work adopts a three-stage pipeline comprising document retrieval, sentence selection, and label prediction, which we detail in this section. ## error analysis we randomly select 200 incorrectly predicted claims by list5 and summarize the most common issues, hoping to identify areas of improvement for future fact verification systems. one common issue is failing to distinguish between similar but semantically different words or phrases. an example of this is the claim "shane mcmahon officially retired on the first day of 2010" to which our document retrieval and sentence selection stages retrieve the sentence "in 2009, mcmahon announced his resignation from wwe which went into effect january 1, 2010". here, retirement and resignation are semantically similar words that both describe individuals leaving their positions. these similarities may have been learned by the pretrained transformer, but it is not always the case that the words imply one another, leading to an incorrect prediction for this claim. another frequent issue is incorrectly labelled claims in the fever dataset, often due to missing evidence in s(q). an example of this is the claim "mickey rourke appeared in a sequel" to which our document retrieval and sentence selection stages retrieve the sentence "since then, rourke has ap-peared in several commercially successful films including the 2010 films iron man 2 and the expendables and the 2011 film immortals". however, the claim was labelled noinfo in the dataset, which is incorrect due to iron man 2 indeed being a sequel. in short, we are bumping into data quality issues in the annotations themselves. ## conclusion in this paper, we present the list5 framework for automated fact verification. list5 consists of a three-stage pipeline -document retrieval, sentence selection, and label prediction. for document retrieval, we combine two strong document retrieval baselines. for sentence selection, we fine-tune a t5 model as a reranker with named entities provided as additional features. for label prediction, we present evidence in a listwise manner to a t5 model, trained on augmented data. our experimental results indicate that list5 achieves the state of the art on the fever task, which we attribute to the framework's ability to reason jointly over multiple pieces of evidence.