Explainable Automated Fact-Checking for Public Health Claims (2020)

Abstract: Fact-checking is the task of verifying the veracity of claims by assessing their assertions against credible evidence. The vast majority of fact-checking studies focus exclusively on political claims. Very little research explores fact-checking for other topics, specifically subject matters for which expertise is required. We present the first study of explainable fact-checking for claims which require specific expertise. For our case study we choose the setting of public health. To support this case study we construct a new dataset PUBHEALTH of 11.8K claims accompanied by journalist crafted, gold standard explanations (i.e., judgments) to support the fact-check labels for claims. We explore two tasks: veracity prediction and explanation generation. We also define and evaluate, with humans and computationally, three coherence properties of explanation quality. Our results indicate that, by training on in-domain data, gains can be made in explainable, automated fact-checking for claims which require specific expertise.

## introduction a great amount of progress has been made in the area of automated fact-checking. this includes more accurate machine learning models for veracity prediction and datasets of both naturally occurring @xcite @xcite and human-crafted @xcite fact-checking claims, against which the models can be evaluated. however, a few blind spots exist in the state-of-the-art. in this work we address specifically two shortcomings: the narrow focus on political claims, and the paucity of explainable systems. one subject area which we believe could benefit from expertise-based fact-checking is public health 1 data and code are available here: https://github. com/neemakot/health-fact-checking -including the study of epidemiology, disease prevention in a population, and the formulation of public policies @xcite . recent events, including the covid-19 pandemic, demonstrate the significant potential harm of misinformation in the public health setting, and the importance in accurately fact-checking claims. unlike political and general misinformation, specific expertise is required in order to fact check claims in this domain. oftentimes this expertise may be limited, and thus claims which surround public health may be inaccessible (e.g., because of the use of jargon and biomedical terminology) in a way political claims are not. nonetheless, like political misinformation, the public health variety is also potentially very dangerous, because it can put people in imminent danger and risk lives. typically, statements which are candidates for fact-checking originate in the political domain @xcite @xcite , and tend to surround more general topics or be non-subject specific @xcite . this follows the trend of the rising interest in political fact-checking in the last decade @xcite . there are on-going efforts with respect to fact-checking scientific claims @xcite . fact-checking in domains where specific subject expertise is required presents an interesting challenge because general purpose fact-checking systems will not necessarily adapt well to these domains. the second shortcoming we look to address is the paucity of explainable models for fact-checking (of any kind). explanations have a particularly important role to play in the task of automated factchecking. the efficacy of journalistic fact-checking hinges on the credibility and reliability of the factcheck, and explanations (e.g., provided by model agnostic tools such as lime @xcite ) can strengthen this by communicating fidelity in predictive models. explainable models can also aid the end users' understanding as they further elucidate claims and their context. in this study we explore the novel case of explainable automated fact-checking for claims for which specialised expertise or in-domain knowledge is essential. for our case study we examine the the public health (biomedical) context. the system for veracity prediction we aim to produce must fulfil two requirements: (1) it should provide a human-understandable explanation (i.e., judgment) for the fact-checking prediction, and (2) that judgement should be understandable for people who do not have expertise in the subject domain. we list the following as our three main contributions in this paper: the explanation model trained on in-domain data outperforms the general purpose model on summarization evaluation and also when evaluated for explanation quality. ## related work a number of recent works in automated factchecking look at various formulations of factchecking and its analogous tasks @xcite @xcite . in this paper, we choose to focus on the two specific aspects of concern to us, which have not been thoroughly explored in the literature. these are domain-specific and expertise-based claim verification and explainability for automated factchecking predictions. ## the pubhealth dataset we constructed a dataset of 11,832 claims for factchecking, which are related a range of health topics including biomedical subjects (e.g., infectious diseases, stem cell research), government healthcare policy (e.g., abortion, mental health, women's health), and other public health-related stories (see unproven, false and mixture examples in to the best of our knowledge, this is the first factchecking dataset to explicitly include gold standard texts provided by journalists specifically as explanation of the fact-checking judgment. we describe below how the data was collected and processed to obtain the final pubhealth dataset, and provide an analysis of the dataset. ## methods in this section we describe in detail the methods we employed for devising automated fact-checking models. we trained two fact-checking models: a classifier for veracity prediction, and a second summarization model for generating fact-checking explanations. the former returns the probability of an input claim text belonging to one of four classes: true, false, unproven, mixture. the latter uses a form of joint extractive and abstractive summarization to generate explanations for the veracity of claims from article text about the claims. full details of hyperparameters chosen and computer infrastructure which was employed can be found in appendix a.2. ## results we conducted experiments to evaluate the performance of both predictor(s) and explainer(s). the performance of the (various incarnations of the) prediction model is evaluated using an automatic approach, whereas the performance of the (two incarnations of the) explainer is assessed using both automatic and human evaluation. ## conclusion and future work in this paper, we explored fact-checking for claims for which specific expertise is required to produce a veracity prediction and explanations (i.e., judg-claim a list of chemicals, written as if they were ingredients on a food label, accurately depicts the chemical composition of a banana. label: true explanation in sum, this graphic accurately depicts the chemicals that comprise a banana, using a variety of tactics to make that completely natural food appear to be full of "chemicals" -something originally created by a high school chemistry teacher as part of a lesson on chemophobia. figure 3 : example of explanation which satisfies all three coherence properties. ments used for awarding the label/veracity prediction). to support this exploration we constructed pubhealth, a sizeable dataset for public health fact-checking and the first fact-checking dataset to include explanations as annotations. our results show that training veracity prediction and explanation generation models on in-domain data improves the accuracy of veracity prediction and the quality of generated explanations compared to training on generic language models without explanation. we hope to explore the topics of explainable fact-checking and specialist fact-checking further. in order to do this, we hope to explore other subjects, in addition to public health, for which factchecking requires a level of expertise in the subject area. furthermore, we hope to explore the quality of fact-checking explanations with respect to properties other than coherence, e.g., actionability and impartiality. lastly, we plan to explore congruity between veracity prediction and explanation generation tasks, i.e., generating explanations which are compatible with the predicted label and vice versa. website explanations ap news (n) leading paragraph. factcheck (f ) summarizing paragraph. fullfact (f ) fact-check conclusions. hnr (r) summary of reliability judment. politifact (f ) fact-check ruling/rating comments. reuters (n) leading paragraph. snopes (f ) fact-check what's true / false / undetermined or concluding paragraph. truthorfict. (f ) summarizing paragraph.