Distilling the Evidence to Augment Fact Verification Models (2020)

Abstract: The alarming spread of fake news in social media, together with the impossibility of scaling manual fact verification, motivated the development of natural language processing techniques to automatically verify the veracity of claims. Most approaches perform a claim-evidence classification without providing any insights about why the claim is trustworthy or not. We propose, instead, a model-agnostic framework that consists of two modules: (1) a span extractor, which identifies the crucial information connecting claim and evidence; and (2) a classifier that combines claim, evidence, and the extracted spans to predict the veracity of the claim. We show that the spans are informative for the classifier, improving performance and robustness. Tested on several state-of-the-art models over the Fever dataset, the enhanced classifiers consistently achieve higher accuracy while also showing reduced sensitivity to artifacts in the claims.

## introduction the increased quantity of information that circulates in social media and on the web every day, together with the high cost of assessing its veracity, has demanded the application of natural language processing (nlp) techniques to the task of fact verification. in the last years, the nlp community has proposed a large number of datasets and approaches for addressing this task, facing complicated challenges that are still far from being solved. the task of fact verification can be split into (i) retrieving one or more candidate pieces of evidence; (ii) assessing whether they are either supporting or refuting a claim, or whether they contains insufficient information to state either of the above. in this paper, we mostly focus on the reasoning between the claim and the evidence. to generate models that work on real world data, fact verification solutions are expected to: (i) per- claim susan sarandon was nominated for five emmy awards. evidence [wiki/susan sarandon] on television, she is a five-time emmy award nominee, including for her guest roles on the sitcoms friends 2001 and malcolm in the middle (2002), and the tv films bernard and doris (2007) and you don't know jack (2010). label support claim fantastic beasts and where to find them was released only in north america on november 18, 2016. evidence [wiki/fantastic beasts and where to find them (film)] fantastic beasts and where to find them premiered in new york city on 10 november 2016 and was released worldwide on 18 november 2016 in 3d, imax 4k laser and other large format cinemas. label refute claim ian brennan is a film screenwriter. evidence [wiki/ian brennan (writer)] ian brennan (born april 23, 1978) is a television writer, actor, producer and director. label not enough information figure 1: examples of claim-evidence pairs from the fever dataset. the evidence spans extracted by our system are underlined and presented in color. to address these two requirements, we propose a model-agnostic framework that includes two modules: (i) a span extractor that aims to identify in the evidence the pieces of relevant information that are informative with respect to the claim; (ii) a classifier that uses the claim, evidence and extracted spans to predict whether the evidence is supporting, refuting or containing insufficient information. the spans extracted by the first module are useful to enhance the classifier and inform the user. humans can in fact exploit the spans to effectively understand why a claim is true or false. we evaluate our pipeline with three highly performing neural models on the fever dataset @xcite , comparing the uninformed to the informed setting. while this dataset includes ground truth for both evidence retrieval and evidence classification, in this paper we only exploit the latter annotations. our experiments show that the models informed with the extracted spans consistently achieve higher performance than their uninformed counterparts, demonstrating the usefulness of spans. we also evaluate our models on the challenging symmetric fever dataset @xcite , which tests system's robustness in absence of fever's artifacts. we find the models trained with our pipeline to achieve higher accuracy. finally, we assess the quality of the extracted spans as decision rationales to be shown to end-user. manually examining a subset of outputs shows that 67% of the support and 88% of the refute spans are well explanatory with respect to the decision, leading to an aggregated score of 75%. ## related work fake news detection has recently gained interest in the nlp community. most of the initial works have focused on style @xcite and linguistic approaches @xcite . despite the good performance in synthetic datasets, these methods failed when applied to real-world data. new approaches based on fact verification over retrieved evidence have therefore taken the stage in the literature. ## method given a claim @xmath0, . . . , c n } and a piece of evidence @xmath1, . . . , e m }, two word sequences of length n and m respectively, the fact verification problem requires to predict the relation @xmath2, (r)efutes, (i)nsufficient} between e and c. framework. we propose a pipeline of two modules: a span extractor m span and a classifier m classifier . the goal of m span (c, e) is to identify polarizing pieces of information {e i 1 , . . . , e i n } in e without which rel(e, c) would be neutral (i.e. c would neither be entailed nor contradicted by e). the identified pieces of information are passed to m classifier , together with c and e, to perform a three-label classification aimed at predicting rel(e, c): m classifier (c, e, {e i 1 , . . . , e i n }) = @xmath3, r, i}. ## experiments we evaluate the three classifiers described in section 3 in two conditions: uninformed (w/o) and informed (with), where the latter refers to the utilization of the information extracted by m span . ## conclusions this paper has introduced a classifier-agnostic framework that allows fact verification models to improve their performance and robustness, utilizing concise spans of the available evidence sentences. the experiments have shown that the extracted spans are indeed informative for the final classifier, supporting the usefulness of the framework. furthermore, this work opens the possibility of providing to the human users a justification for the model's predictions.