Generating Fact Checking Explanations (2020)

Abstract: Most existing work on automated fact checking is concerned with predicting the veracity of claims based on metadata, social network spread, language used in claims, and, more recently, evidence supporting or denying claims. A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process – generating justifications for verdicts on claims. This paper provides the first study of how these explanations can be generated automatically based on available claim context, and how this task can be modelled jointly with veracity prediction. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system. The results of a manual evaluation further suggest that the informativeness, coverage and overall quality of the generated explanations are also improved in the multi-task model.

## introduction when a potentially viral news item is rapidly or indiscriminately published by a news outlet, the responsibility of verifying the truthfulness of the item is often passed on to the audience. to alleviate this problem, independent teams of professional fact checkers manually verify the veracity and credibility of common or particularly checkworthy statements circulating the web. however, these teams have limited resources to perform manual fact checks, thus creating a need for automating the fact checking process. the current research landscape in automated fact checking is comprised of systems that estimate the veracity of claims based on available metadata and evidence pages. datasets like liar @xcite and the multi-domain dataset mul-tifc @xcite provide real-world claim: the last major oil spill from a drilling accident in america happened over 40 years ago in 1969. ruling comments: (...) the last major oil spill from a drilling accident in america happened over 40 years ago in 1969. justification: while the nation's largest oil well blowout did take place in 1969, it's not factually correct to call it the "last major oil spill". first of all, two of the largest blowouts in the world took place in the u. s. the following year. more importantly, experts agree that spills far smaller in volume to the 1969 disaster have been devastating. from a scientific perspective, johnson's decision to single out the 1969 blowout as the last "major" one makes no sense. ruling: half-true a prevalent component of existing fact checking systems is a stance detection or textual entailment model that predicts whether a piece of evidence contradicts or supports a claim @xcite @xcite . existing research, however, rarely attempts to directly optimise the selection of relevant evidence, i.e., the self-sufficient explanation for predicting the veracity label @xcite . on the other hand, @xcite have reported a significant performance improvement of over 10% macro f1 score when the system is provided with a short human explanation of the veracity label. still, there are no attempts at automatically producing explanations, and automating the most elaborate part of the processproducing the justification for the veracity prediction -is an understudied problem. in the field of nlp as a whole, both explainability and interpretability methods have gained importance recently, because most state-of-the-art models are large, neural black-box models. interpretability, on one hand, provides an overview of the inner workings of a trained model such that a user could, in principle, follow the same reasoning to come up with predictions for new instances. however, with the increasing number of neural units in published state-of-the-art models, it becomes infeasible for users to track all decisions being made by the models. explainability, on the other hand, deals with providing local explanations about single data points that suggest the most salient areas from the input or are generated textual explanations for a particular prediction. saliency explanations have been studied extensively @xcite @xcite , however, they only uncover regions with high contributions for the final prediction, while the reasoning process still remains behind the scenes. an alternative method explored in this paper is to generate textual explanations. in one of the few prior studies on this, the authors find that feeding generated explanations about multiple choice question answers to the answer predicting system improved qa performance @xcite . inspired by this, we research how to generate explanations for veracity prediction. we frame this as a summarisation task, where, provided with elaborate fact checking reports, later referred to as ruling comments, the model has to generate veracity explanations close to the human justifications as in the example in ## dataset existing fact checking websites publish claim veracity verdicts along with ruling comments to support the verdicts. most ruling comments span over long pages and contain redundancies, making them hard to follow. textual explanations, by contrast, are succinct and provide the main arguments behind the decision. politifact foot_0 provides a summary of a claim's ruling comments that summarises the whole explanation in just a few sentences. we use the politifact-based dataset liar-plus @xcite , which contains 12,836 statements with their veracity justifications. the justifications are automatically extracted from the long ruling comments, as their location is clearly indicated at the end of the ruling comments. any sentences with words indicating the label, which @xcite select to be identical or similar to the label, are removed. we follow the same procedure to also extract the ruling comments without the summary at hand. we remove instances that contain fewer than three sentences in the ruling comments as they indicate short veracity reports, where no summary is present. the final dataset consists of 10,146 training, 1,278 validation, and 1,255 test data points. a claim's ruling comments in the dataset span over 39 sentences or 904 words on average, while the justification fits in four sentences or 89 words on average. ## method we now describe the models we employ for training separately (1) an explanation extraction and (2) veracity prediction, as well as (3) the joint model trained to optimise both. the models are based on distilbert @xcite , which is a reduced version of bert @xcite performing on par with it as reported by the authors. for each of the models described below, we take the version of distilbert that is pre-trained with a language-modelling objective and further fine-tune its embeddings for the specific task at hand. ## automatic evaluation we first conduct an automatic evaluation of both the veracity prediction and veracity explanation models. ## manual evaluation as the rouge score only accounts for word-level similarity between gold and predicted justifications, we also conduct a manual evaluation of the quality of the produced veracity explanations. ## conclusions we presented the first study on generating veracity explanations, and we showed that veracity prediction can be combined with veracity explanation generation and that the multi-task set-up improves the performance of the veracity system. a manual evaluation shows that the coverage and the overall quality of the explanation system is also improved in the multi-task set-up. for future work, an obvious next step is to investigate the possibility of generating veracity explanations from evidence pages crawled from the web. furthermore, other approaches of generating veracity explanations should be investigated, especially as they could improve fluency or decrease the redundancy of the generated text.