Evidence-based Fact-Checking of Health-related Claims (2021)

Abstract: The task of verifying the truthfulness of claims in textual documents, or fact-checking, has received significant attention in recent years. Many existing evidence-based factchecking datasets contain synthetic claims and the models trained on these data might not be able to verify real-world claims. Particularly few studies addressed evidence-based fact-checking of health-related claims that require medical expertise or evidence from the scientific literature. In this paper, we introduce HEALTHVER, a new dataset for evidence-based fact-checking of health-related claims that allows to study the validity of real-world claims by evaluating their truthfulness against scientific articles. Using a three-step data creation method, we first retrieved real-world claims from snippets returned by a search engine for questions about COVID-19. Then we automatically retrieved and re-ranked relevant scientific papers using a T5 relevance-based model. Finally, the relations between each evidence statement and the associated claim were manually annotated as SUPPORT, REFUTE and NEUTRAL. To validate the created dataset of 14,330 evidence-claim pairs, we developed baseline models based on pretrained language models. Our experiments showed that training deep learning models on real-world medical claims greatly improves performance compared to models trained on synthetic and open-domain claims. Our results and manual analysis suggest that HEALTHVER provides a realistic and challenging dataset for future efforts on evidence-based fact-checking of health-related claims. The dataset, source code, and a leaderboard are available at https://github.com/sarrouti/healthver .

## introduction the exponential growth of textual information in the form of news, forums, and stories on the web has resulted in the explosion of misinformation @xcite 2020). while false information could be dangerous in general, medical misinformation, in particular, presents a challenge to human health and could be detrimental when search engines or social media are used to guide health-related decisions @xcite . for instance, the covid-19 pandemic has caused the spread of false claims about the origin, prevention, diagnosis, and treatment of the disease (naeem et al., 2020). covid-19 related misinformation caused people to turn to fake and unproven cures @xcite . recently, retrieving and debunking misinformation has received significant attention, especially from fact-checking organizations (e.g. snopes) that debunk false information. false claims and fake news stories, however, are still spreading on the web @xcite . while most factchecking organizations use human validation of information, the ever-increasing amount of new information on the internet makes manual verification challenging, time-consuming, and costly (rashkin et al., 2017; thorne and vlachos, 2018; @xcite . moreover, in contrast to the open domain, fact-checking medical information requires deep medical knowledge about the topic of the claims @xcite . as a result, automated fact-checking tools for verifying the veracity of a given health-related claim or news story are needed to assist the information seekers in evaluating the retrieved information from uncontrolled data collections such as the web. automated fact-checking systems based on deep learning to identify misinformation distributed on the internet is a promising approach to counter its spread. evaluating the veracity of a given claim against textual sources (e.g. scientific articles, wikipedia) that can support, refute or relate to the claim has been explored to fight the spread of misinformation @xcite . the claims on which the existing fact-checking systems rely for training are synthetic, since they were manually created from the sentences or citations retrieved from a corpus of documents. for instance, claims in the fever dataset @xcite were manually created by mutating sentences from wikipedia documents. whereas the scientific claims in @xcite were created from citation sentences by annotators. "natural" claims, or real-world claims, expressed by internet users differ from the manually created claims for several reasons. recent events such as the covid-19 pandemic showed that real-world claims could include multiple facts (e.g. "vitamins c and d may help your immune system fight covid-19") which makes the verification process more complicated as the evidence (i.e. a scientific article) may support the claim about one fact and refute another one that is stated in the same claim. moreover, most of the claims that spread on the web use speculative and vague language (e.g. "social distancing measures could be effective in stopping the spread of the virus"). therefore, the deep learning models trained on the manually created claims are unlikely to be able to validate claims found on the web. furthermore, verifying claims in domains such as the medical domain, where medical expertise is needed, makes the task more challenging. the adaptation of factchecking models trained on open-domain claims to health-related claims might not necessarily work well. to tackle the aforementioned issues, we introduce a new fact-checking dataset called healthver, a new dataset for evidence-based fact-checking of health-related claims based on scientific articles. compared to the existing efforts, we use naturally-occurring claims from the web and scientific articles for verification. as shown in figure 2 , given a claim and a relevant scientific article retrieved from a corpus of scientific articles, our verification system predicts three types of relations between the claim and the evidence extracted from an article: supports, refutes, and neutral. in summary, this paper makes the following main contributions: 1. we introduce healthver, a new manually annotated dataset consisting of 14,330 evidence-claim pairs with their veracity label (i.e. supports, re-futes, and neutral). to the best of our knowledge, this is the first evidence-based fact-checking study that investigates the veracity of real-world claims against scientific articles. ## related work in recent years, there have been growing concerns about the rampant spread of fake news, false claims, and fabricated stories @xcite @xcite . due to the proliferation of misinformation, several efforts have been made to construct fact-checking datasets to advance automated fact-checking systems @xcite al., 2019; thorne et al., 2021; schuster et al., 2021). vlachos and riedel (2014) collected a claim-verification dataset from factchecking websites (e.g. politifact) that contains a limited number of claims (106 claims). wang (2017) constructed a large dataset of 12.8k claims from the politifact website. nakov et al. (2018) introduced a dataset for the clef-2018 checkthat! shared task on political debates in english and arabic. a common feature for all the aforecited datasets is that they only contain claims without evidence to support or refute them. for evidence-based fact-checking, thorne et al. (2018) constructed the fever dataset of 185.4k claims generated by mutating sentences extracted from wikipedia. however, the claims are synthetic since they are created by altering the evidence sentences. augenstein et al. (2019) introduced multifc, the claim verification dataset of natural claims. it consists of 34,918 claims, collected from 26 fact checking websites in english, the evidence pages to verify the claims, and other metadata information. in the medical domain, a new dataset was introduced for the trec 2020 health misinformation track. documents related to covid-19 from the commoncrawl news dataset 1 have been used. in this dataset, the evidence for claim validation was missing. kotonya and toni (2020) built a dataset called pubhealth which includes 11.8k claims accompanied by journalists' explanations from factchecking websites (e.g. snopes, politifact). pub-health is designed to evaluate veracity prediction and explanation generation tasks. the majority of the claims in this dataset are false. wadden et al. (2020) created scifact, a corpus of 1.4k scientific claims accompanied by abstracts that support or refute each claim. this dataset, however, contains synthetic claims. the existing datasets for evidence-based factchecking systems are either based on mutated sentences (e.g. creating claims from wikipedia and citations sentences), that are not real-world and natural claims, or use journalists' explanations from fact-checking websites. claims found on the internet are arguably more complicated and challenging to verify than synthetic claims. therefore, to mitigate the above discrepancies between the real-world claims and the training data, we introduce a new dataset of real-world claims related to covid-19 with associated evidence extracted from scientific articles, manually annotated with three types of relations: supports, refutes, 1 from january 1st, 2020 to april 30, 2020. and neutral. ## the healthver dataset this section describes our proposed approach to create the healthver dataset, which consists of three main stages: 1. claim retrieval: retrieving, extracting, and selecting real-world health-related claims from snippets returned by a search engine for given questions that are asked online. 2. evidence retrieval: automatically retrieving scientific articles relevant to these claims and then manually extracting evidence from their abstracts. ## dataset validation we validate healthver and its ability to support the fact-checking task compared to the existing datasets. ## conclusion in this paper, we explored evidence-based factchecking of real-world and health-related claims found on the internet. to this end, we introduced healthver, a new evidence-based fact-checking dataset, to verify the veracity of real-world claims by evaluating their assertions against scientific articles. we analyzed the complexity of the claims in healthver using a relation extraction ensemble and compared its generalization potential with existing datasets using pairwise zero-shot accuracy deltas and a new measure that takes into account training set sizes. we found that the proportion of complex claims in healthver is consistently higher than in the existing datasets at all complexity levels. our experiments showed that training factchecking models on real-world claims improves the accuracy of these models compared to training on synthetic claims. the results also showed that training models on in-domain data substantially improves health-related claim verification accuracy compared to training on open-domain data. we believe that healthver will provide a realistic and challenging testbed for new evidence-based fact-checking systems for real-world claims.