C rowd C hecked: Detecting Previously Fact-Checked Claims in Social Media (2022)

Abstract: While there has been substantial progress in developing systems to automate fact-checking, they still lack credibility in the eyes of the users. Thus, an interesting approach has emerged: to perform automatic fact-checking by verifying whether an input claim has been previously fact-checked by professional fact-checkers and to return back an article that explains their decision. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times. Yet, a major issue when building such systems is the small number of known tweet–verifying article pairs available for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a corresponding fact-checking article. We further propose an end-to-end framework to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on the CLEF’21 CheckThat! test set show improvements over the state of the art by two points absolute. Our code and datasets are available at https://github.com/mhardalov/crowdchecked-claims

## introduction the massive spread of disinformation online, especially in social media, was counter-acted by major efforts to limit the impact of false information not only by journalists and fact-checking organizations but also by governments, private companies, researchers, and ordinary internet users. this includes building systems for automatic factchecking @xcite @xcite @xcite , fake news @xcite , and fake news website detection @xcite @xcite . unfortunately, fully automatic systems still lack credibility, and thus it was proposed to focus on detecting previously fact-checked claims instead: given a user comment, detect whether the claim it makes was previously fact-checked with respect to a collection of verified claims and their corresponding articles (see we propose to bridge this gap using crowd factchecking to create a large collection of tweetverifying article pairs, which we then label (if the pair is correctly matched) automatically using distant supervision. an example is shown in figure 1 . our contributions are as follows: ## experiments in this section, we describe our experimental setup, baselines, and experimental results. the training procedure and the hyper-parameters are described in more detail in appendix a. ## discussion our proposed distant supervision data selection strategies show promising results, achieving sota results on the checkthat '21. nonetheless, we are not able to identify all matching pairs in the list of candidates in crowdchecked. hereby, we try to estimate their number using statistics from our manual annotations, foot_1 as shown in tables 3 and 4 . in particular, we estimate it by multiplying the fraction of correct pairs in each similarity bin by the number of examples in this bin. based on cosine similarity, we estimate that out of the 332,600 pairs, the matching pairs are approximately 90,170 (27.11%). based on the jaccard distribution, we estimate that 14.79% of all tweet-conversations (root of the conversation), and 22.23% of the tweet-reply (the tweet before the current one in the conversation) pairs are good, or nearly 61,500 examples. our experiments show that the models can effectively account for the noise in the training data. the self-adaptive training and the additional weighing in the loss (described in section 3) yield 1 additional map@5 point each. this suggests that learning from noisy labels @xcite @xcite and using all examples in crowdchecked can improve the results even further. moreover, incorporating the negative examples (non-matching pairs) from crowdchecked in the training could also help @xcite . recently, the task received more attention. @xcite collected two datasets, from politi-fact (political debates) and snopes (tweets), of claims and corresponding fact-checking articles. the clef checkthat! lab @xcite @xcite ) extended these datasets with more data in english and arabic. the best systems @xcite @xcite used a combination of bm25 retrieval, semantic similarity using embeddings @xcite , and reranking. @xcite used extra data from fact-checking datasets @xcite @xcite . finally, @xcite and @xcite explored the role of the context in detecting previously fact-checked claims in political debates. our work is most similar to that of @xcite , who mined 19k tweets and corresponding fact-checked articles. unlike them, we focus on textual claims (they were interested in multimodal tweets with images), we collect an order of magnitude more examples, and we propose a novel approach to learn from such noisy data directly (while they manually checked each example). training with noisy data leveraging large collections of unlabeled data has been at the core of large-scale language models using transformers @xcite , such as gpt @xcite , bert @xcite , and roberta @xcite . recently, such models used noisy retrieved data @xcite or active relabeling and data augmentation @xcite . distant supervision is also a crucial part of recent breakthroughs in few-shot learning @xcite ). yet, there has been little work of using noisy data for fact-checking tasks. @xcite collected tweets containing a link to a fact-checking website, based on which they tried to learn a factchecking language and to generate automatic answers. @xcite used similar data from tweets for fact-checking url recommendations. unlike the above work, here we propose an automatic procedure for labeling and self-training specifically designed for the task of detecting previously fact-checked claims. ## conclusion and future work we presented crowdchecked, a large dataset for detecting previously fact-checked claims, with more than 330,000 pairs of tweets and corresponding fact-checking articles posted by crowd factcheckers. we further investigated two techniques for labeling the data using distance supervision, resulting in training sets of 3.5k-50k examples. we also proposed an approach for training from noisy data using self-adaptive learning and additional weights in the loss function. furthermore, we demonstrated that our data yields sizable performance gains of four points in terms mrr, p@1, and map@5 over strong baselines. finally, we demonstrated improvements over the state of the art on the checkthat '21 test set by two points, when using our proposed dataset and pipeline. in future work, we plan to experiment with more languages and more distant supervision techniques such as predictions from an ensemble model. our models can help fight the covid-19 infodemic, and they could support analysis and decision-making for the public good. however, the models can also be misused by malicious actors. therefore, we ask the users to be aware of potential misuse. with the possible ramifications of a highly subjective dataset, we distribute it for research purposes only, without a license for commercial use. any biases found in the dataset are unintentional, and we do not intend to do harm to any group or individual.