Extractive and Abstractive Explanations for Fact-Checking and Evaluation of News (2021)

Abstract: In this paper, we explore the construction of natural language explanations for news claims, with the goal of assisting fact-checking and news evaluation applications. We experiment with two methods: (1) an extractive method based on Biased TextRank – a resource-effective unsupervised graph-based algorithm for content extraction; and (2) an abstractive method based on the GPT-2 language model. We perform comparative evaluations on two misinformation datasets in the political and health news domains, and find that the extractive method shows the most promise.

## introduction navigating the media landscape is becoming increasingly challenging given the abundance of misinformation, which reinforces the importance of keeping our news consumption focused and informed. while fake news and misinformation have been a recent focus of research studies @xcite @xcite , the majority of this work aims to categorize claims, rather than generate explanations that support or deny them. this is a challenging problem that has been mainly tackled by expert journalists who manually verify the information surrounding a given claim and provide a detailed verdict based on supporting or refuting evidence. more recently, there has been a growing interest in creating computational tools able to assist during this process by providing supporting explanations for a given claim based on the news content and context @xcite . while a true or false veracity label does not provide enough information and a detailed fact-checking report or news article might take long to read, bitesized explanations can bridge this gap and improve the transparency of automated news evaluation systems. to contribute to this line of work, our paper explores two approaches to generate supporting explanations to assist with the evaluation of news. first, we investigate how an extractive method based on biased textrank @xcite ) can be used to generate explanations. second, we explore an abstractive method based on gpt-2, a large generative language model @xcite . our methods take as input a news article and a claim and generate a claim-focused explanation by extracting or generating relevant information to the original article in relation to the claim. we evaluate our proposed methods on the health care and political domains, where misinformation is abundant. as current news on the covid-19 pandemic and the elections are overloading social media outlets, we find these domains to be of timely importance. through comparative experiments, we find that both methods are effective at generating explanations for news claims, with the extractive approach showing the most promise for this task. ## related work while explainability in ai has been a central subject of research in recent years @xcite @xcite , the generation of natural language explanations is still relatively understudied. @xcite propose e-snli, a natural language (nl) inference dataset augmented with human-annotated nl explanations. in their paper, camburu et al. generated nl explanations for premise and hypothesis pairs for an inference task using the in-fersent @xcite architecture. @xcite propose the task of generating "faithful" (i.e., aligned with the model's internal decision making) nl explanations and propose nile, a method that jointly produces nli labels and faithful nl explanations. generating explanations in the context of news and fact-checking is a timely and novel topic in the claim: nearly half of oregon's children are poor. fact-check report: ...jim francesconi...said..."nearly half of oregon's children are poor." he said the information came from a 2012 report...according to that report, "nearly 50% of children are either poor or lowincome." francesconi almost immediately realized his mistake. "in retrospect, i wish i would have said poor or low income."...there is a distinction between poor and low income as far as the u.s. government is concerned." ... if you check the...census information, you'll find that...23 percent of children in oregon live in...below...poverty level while another 21 percent live in low-income families. as far as the u.s. government is concerned, about a quarter of the state's children are poor, not half... (redacted) supporting explanation (ground truth): so where does this leave us? francesconi said in an opinion piece that "nearly half of oregon's children are poor." in fact, if you use federal definitions for poverty, about a quarter are poor and another quarter are low-income. but experts tell us that families that are described as low-income still struggle to meet their basic needs and, for all intents and purposes, qualify as poor. be that as it may, francesconi was referencing a report that used the federal definitions. biased textrank (extractive): "nearly half of oregon's children are poor." according to that report, "nearly 50% of children are either poor or low-income." low income refers to families between 100 and 200 percent of the federal poverty level. as far as the u. nlp community @xcite @xcite . in @xcite the authors proposed a supervised bert @xcite based model for jointly predicting the veracity of a claim by extracting supporting explanations from fact-checked claims in the liar-plus (alhindi et al., 2018) dataset. @xcite constructed a dataset for a similar task in the public health domain and provided baseline models for explainable fact verification using this dataset. @xcite used explanations about a claim to assist fact-checkers and showed that explanations improved both the efficiency and the accuracy of the fact-checking process. ## methods we explore two methods for producing natural language explanations: an extractive unsupervised method based on biased textrank, and an abstractive method based on gpt-2. ## experimental results as results in ## discussion our evaluations indicate that biased textrank shows the most promise, while the gpt-2 based model mostly follows in performance. keeping in mind that the gpt-2 based model is solving the harder problem of generating language, it is worth noting the little supervision it receives on both datasets, especially on the hnr dataset where the average size of the training data is 849. in terms of resource efficiency and speed, biased textrank is faster and lighter than the gpt-2 based model. excluding the time needed to fine-tune the gpt-2 model, it takes approximately 60 seconds on a gpu to generate a coherent abstractive explanation on average on the liar-plus dataset, while biased textrank extracts explanations in the order of milliseconds and can even do it without a gpu in a few seconds. we find biased textrank's efficiency as another advantage of the unsupervised algorithm over the gpt-2 based model. ## conclusion in this paper, we presented extractive and abstractive methods for generating supporting explanations for more convenient and transparent human consumption of news. we evaluated our methods on two domains and found promising results for producing explanations. in particular, biased text-rank (an extractive method) outperformed the unsupervised baselines on the liar-plus dataset and performed reasonably close to the extractive ground-truth on the hnr dataset. for future work, we believe generating abstractive explanations should be a priority, since intuitively an increase in the readability and coherence of the supporting explanations will result in improvements in the delivery and perception of news.