That is a Known Lie: Detecting Previously Fact-Checked Claims (2020)

Abstract: The recent proliferation of ”fake news” has triggered a number of responses, most notably the emergence of several manual fact-checking initiatives. As a result and over time, a large number of fact-checked claims have been accumulated, which increases the likelihood that a new claim in social media or a new statement by a politician might have already been fact-checked by some trusted fact-checking organization, as viral claims often come back after a while in social media, and politicians like to repeat their favorite statements, true or false, over and over again. As manual fact-checking is very time-consuming (and fully automatic fact-checking has credibility issues), it is important to try to save this effort and to avoid wasting time on claims that have already been fact-checked. Interestingly, despite the importance of the task, it has been largely ignored by the research community so far. Here, we aim to bridge this gap. In particular, we formulate the task and we discuss how it relates to, but also differs from, previous work. We further create a specialized dataset, which we release to the research community. Finally, we present learning-to-rank experiments that demonstrate sizable improvements over state-of-the-art retrieval and textual similarity approaches.

## introduction the year 2016 was marked by massive disinformation campaigns related to brexit and the us presidential elections. while false statements are not a new phenomenon, e.g., yellow press and tabloids have been around for decades, this time things were notably different in terms of scale and effectiveness thanks to social media platforms, which provided both a medium to reach millions of users and an easy way to micro-target specific narrow groups of voters based on precise geographical, demographic, psychological, and/or political profiling. governments, international organizations, tech companies, media, journalists, and regular users launched a number of initiatives to limit the impact of the newly emerging large-scale weaponization of disinformation foot_0 online. notably, this included manual fact-checking initiatives, which aimed at debunking various false claims, with the hope to limit its impact, but also to educate the public that not all claims online are true. over time, the number of such initiatives grew substantially, e.g., at the time of writing, the duke reporters' lab lists 237 active fact-checking organizations plus another 92 inactive. 2 while some organizations debunked just a couple of hundred claims, others such as politifact, 3 factcheck.org, 4 snopes, 5 and full fact 6 have fact-checked thousands or even tens of thousands of claims. the value of these collections of resources has been recognized in the research community, and they have been used to train systems to perform automatic fact-checking @xcite @xcite or to detect checkworthy claims in political debates @xcite @xcite . there have also been datasets that combine claims from multiple fact-checking organizations @xcite , again with the aim of performing automatic fact-checking. it has been argued that checking against a database of previously fact-checked claims should be an integral step of an end-to-end automated factchecking pipeline @xcite . this is illustrated in figure 1 , which shows the general steps of such a pipeline @xcite : (i) assess the check-worthiness of the claim (which could come from social media, from a political debate, etc.), (ii) check whether a similar claim has been previously fact-checked (the task we focus on here), (iii) retrieve evidence (from the web, from social media, from wikipedia, from a knowledge base, etc.), and (iv) assess the factuality of the claim. from a fact-checkers' point of view, the abundance of previously fact-checked claims increases the likelihood that the next claim that needs to be checked would have been fact-checked already by some trusted organization. indeed, viral claims often come back after a while in social media, and politicians are known to repeat the same claims over and over again. 7 thus, before spending hours fact-checking a claim manually, it is worth first making sure that nobody has done it already. on another point, manual fact-checking often comes too late. a study has shown that "fake news" spreads six times faster than real news @xcite . another study has indicated that over 50% of the spread of some viral claims happens within the first ten minutes of their posting on social media @xcite . at the same time, detecting that a new viral claim has already been fact-checked can be done automatically and very quickly, thus allowing for a timely action that can limit the spread and the potential malicious impact. from a journalistic perspective, the ability to check quickly whether a claim has been previously fact-checked could be revolutionizing as it would allow putting politicians on the spot in real time, e.g., during a live interview. in such a scenario, automatic fact-checking would be of limited utility as, given the current state of technology, it does not offer enough credibility in the eyes of a journalist. interestingly, despite the importance of the task of detecting whether a claim has been fact-checked in the past, it has been largely ignored by the research community. here, we aim to bridge this gap. our contributions can be summarized as follows: the remainder of this paper is organized as follows: section 2 discusses related work, section 3 introduces the task, section 4 presents the dataset, section 5 discusses the evaluation measures, section 6 presents the models we experiment with, section 7 described our experiments, and section 8 concludes and discusses future work. ## related work to the best of our knowledge, the task of detecting whether a claim has been previously fact-checked was not addressed before. @xcite mentioned it as an integral step of their end-to-end automated fact-checking pipeline, but there was very little detail provided about this component and it was not evaluated. in an industrial setting, google has developed fact check explorer, foot_5 which is an exploration tool that allows users to search a number of factchecking websites (those that use claimreview from schema.org foot_6 ) for the mentions of a topic, a person, etc. however, the tool cannot handle a complex claim, as it runs google search, which is not optimized for semantic matching of long claims. while this might change in the future, as there have been reports that google has started using bert in its search, at the time of writing, the tool could not handle a long claim as an input. a very similar work is the claimskg dataset and system @xcite , which includes 28k claims from multiple sources, organized into a knowledge graph (kg). the system can perform data exploration, e.g., it can find all claims that contain a certain named entity or keyphrase. in contrast, we are interested in detecting whether a claim was previously fact-checked. other work has focused on creating datasets of textual fact-checked claims, without building kgs. some of the larger ones include the liar, liar dataset of 12.8k claims from politifact @xcite , and the multifc dataset of 38k claims from 26 fact-checking organizations @xcite , the 10k claims truth of various shades @xcite dataset, among several other datasets, which were used for automatic fact-checking of individual claims, not for checking whether an input claim was fact-checked previously. note that while the above work used manually normalized claims as input, we work with naturally occurring claims as they were made in political debates and speeches or in social media. there has also been a lot of research on automatic fact-checking of claims and rumors, going in several different directions. one research direction focuses on the social aspects of the claim and how users in social media react to it @xcite @xcite @xcite . another direction mines the web for information that proves or disproves the claim @xcite @xcite @xcite . in either case, it is important to model the reliability of the source as well as the stance of the claim with respect to other claims; in fact, it has been proposed that a claim can be fact-checked based on its source alone @xcite or based on its stance alone @xcite . a third direction performs fact-checking against wikipedia @xcite , or against a general collection of documents @xcite . a fourth direction uses a knowledge base or a knowledge graph @xcite @xcite . yet another direction performs factchecking based on tables @xcite . there is also recent work on using language models as knowledge bases @xcite . ours is yet another research direction. while our main contribution here is the new task and the new dataset, we should also mentioned some work on retrieving documents. in our experiments, we perform retrieval using bm25 @xcite and re-ranking using bert-based similarity, which is a common strategy in recent state-of-the-art retrieval models @xcite @xcite . our approach is most similar to that of (akkalyoncu yilmaz et al., 2019a), but we differ, as we perform matching, both with bm25 and with bert, against the normalized claim, against the title, and against the full text of the articles in the fact-checking dataset; we also use both scores and reciprocal ranks when combining different scores and rankings. moreover, we use sentence-bert instead of bert. previous work has argued that bert by itself does not yield good sentence representation. thus, approaches such as sentence-bert (reimers and gurevych, 2019) have been proposed, which are specifically trained to produce good sentence-level representations. this is achieved using siamese bert networks that are fine-tuned on nli and sts-b data. indeed, in our experiments, we found sentence-bert to perform much better than bert. the universal sentence encoder @xcite is another alternative, but sentence-bert worked better in our experiments. finally, our task is related to semantic relatedness tasks, e.g., from the glue benchmark @xcite , such as natural language inference, or nli task @xcite , recognizing textual entailment, or rte @xcite , paraphrase detection @xcite , and semantic textual similarity, or sts-b @xcite . however, it also differs from them, as we will see in the following section. ## task definition we define the task as follows: given a checkworthy input claim and a set of verified claims, rank those verified claims, so that the claims that can help verify the input claim, or a sub-claim in it, are ranked above any claim that is not helpful to verify the input claim. input claim manually annotated claim in politifact 1 richard nixon released tax returns when he was under audit. richard nixon released tax returns when he was under audit. 2 hillary wants to give amnesty. ## datasets we created two datasets by collecting, for each of them, a set of verified claims and matching input-verified claims pairs (below, we will also refer to these pairs as input-verclaim pairs): the first dataset, politifact, is about political debates and speeches and it is described in section 4.1; the second dataset, snopes, includes tweets, and it is described in section 4.2. ## evaluation measures we treat the task as a ranking problem. thus, we use ranking evaluation measures, namely mean reciprocal rank (mrr), mean average precision (map), and map truncated to rank k (map@k). we also report haspositive@k, i.e., whether there is a true positive among the top-k results. measures such as map@k and haspositive@k for @xmath0, 3, 5} would be relevant in a scenario, where a journalist needs to verify claims in real time, in which case the system would return a short list of 3-5 claims that the journalist can quickly skim and make sure they are indeed a true match. we further report map@k and haspositive@k for @xmath1, 20} as well as map (untruncated), which would be more suitable in a non-real-time scenario, where recall would be more important. ## models here, we describe the models we experiment with. ## experiments below we describe our experiments on the poli-tifact and the snopes datasets. we start with irbased models, followed by bert-based semantic similarity on claims and articles, and finally we experiment with pairwise learning-to-rank models. ## conclusions and future work we have argued for the need to address detecting previously fact-checked claims as a task of its own right, which could be an integral part of automatic fact-checking, or a tool to help human fact-checkers or journalists. we have created specialized datasets, which we have released, together with our code, to the research community in order to enable further research. finally, we have presented learning-torank experiments, demonstrating sizable improvements over state-of-the-art retrieval and textual similarity approaches. in future work, we plan to extend this work to more datasets and to more languages. we further want to go beyond textual claims, and to take claimimage and claim-video pairs as an input.