Multilingual Previously Fact-Checked Claim Retrieval (2023)

Abstract: Fact-checkers are often hampered by the sheer amount of online content that needs to be fact-checked. NLP can help them by retrieving already existing fact-checks relevant to the content being investigated. This paper introduces a new multilingual dataset for previously fact-checked claim retrieval. We collected 28k posts in 27 languages from social media, 206k fact-checks in 39 languages written by professional fact-checkers, as well as 31k connections between these two groups. This is the most extensive and the most linguistically diverse dataset of this kind to date. We evaluated how different unsupervised methods fare on this dataset and its various dimensions. We show that evaluating such a diverse dataset has its complexities and proper care needs to be taken before interpreting the results. We also evaluated a supervised fine-tuning approach, improving upon the unsupervised method significantly.

## introduction fact-checking organizations have made progress in recent years in manually and professionally factchecking viral content @xcite . to reduce some of the fact-checkers' manual efforts and make their work more effective, several studies have recently examined their needs and identified tasks that could be automated @xcite @xcite @xcite . these tasks include searching for the source of evidence for verification, searching for other versions of misinformation, and searching within existing factchecks. they were identified as particularly challenging for fact-checkers working in low-resource languages @xcite . in this work, we focus on previously fact-checked claim retrieval (pfcr) @xcite . given a text making an input claim (e.g., a social media post) and a set of fact-checked claims, our task is to rank the fact-checked claims so that those that are the most relevant w.r.t. the input claim (and thus the most useful from the fact-checker's perspective) are ranked as high as possible. previously, this task was mostly done in english. other languages that have been considered include arabic @xcite , bengali, hindi, malayalam, and tamil @xcite . however, many other languages or even entire major language families have not been considered at all. additionally, so far only monolingual pfcr has been tackled, when the input claim and the fact-checked claims are in the same language. to address these shortcomings, we introduce in this paper a new extensive multilingual dataset. our two main contributions are: all the posts were previously reviewed by professional fact-checkers who also assigned appropriate fact-checks to them. we collected these assignments and gathered 31,305 pairs consisting of a post and a fact-check reviewing the claim made in the post. 4,212 of these pairs are crosslingual (i.e., the language of the fact-check and the language of the post are different). this dataset introduces crosslingual pfcr as a new task that has not been tackled before. this is the biggest collection of such pairs that were confirmed by professional fact-checkers. the dataset also includes ocr transcripts of the images attached to the posts and machine translation of all the data into english. ## related work other names are used for pfcr or similar tasks for various reasons, e.g., fact-checking url recommendation (vo and lee, 2018), fact-checked claims detection @xcite , verified claim retrieval @xcite , searching for fact-checked information (vo and lee, 2020), or claim matching @xcite . datasets. checkthat! datasets @xcite have the most similar collection approach to ours. they collect english and arabic tweets mentioned in fact-checks to create preliminary pairs and then manually filter them. compared to this work, we broaden the scope of data collection and omit the manual cleaning in favor of using fact-checkers' reports. @xcite collected data from fact-checking of english political debates done by fact-checkers. the crowdchecked dataset @xcite was created by searching for fact-check urls on twitter and collecting english tweets from retrieved threads. the process is inherently noisy and, the authors propose different noise filtering techniques. @xcite collected several million chat messages from public chat groups and tiplines in english, bengali, hindi, malayalam, and tamil and 150k fact-checks. then they sampled roughly 2,300 pairs based on their embedding similarity and manually annotated them. in the end, they obtained only roughly 250 positive pairs. @xcite matched covid-19 tweets and 90 covid-19 claims in a similar manner. their data could be used for pfcr, but the authors worked on classification instead. pfcr datasets are summarized in methods. methods used for pfcr are usually either bm25 (and other similar information retrieval algorithms) or various text embedding-based approaches @xcite . reranking is often used to combine several methods to side-step compute requirements or as a sort of ensembling @xcite . pfcr task is also a target of the clef's checkthat! challenge, with many teams contributing with their solutions @xcite . other methods use visual information from images @xcite , abstractive summarization @xcite , or key sentence identification @xcite to improve the results. ## our dataset our dataset multiclaim consists of fact-checks, social media posts and pairings between them. ## unsupervised evaluation we formulate the task we are solving with our dataset as a ranking task, i.e., for each post, the methods rank all the fact-checks. then, we evaluate the performance based on the rank of the desired fact-checks by using success-at-k (s@k) as the main evaluation metric. we define as the percentage of pairs when the desired fact-check ends up in the top k. throughout the paper, we report this metric with the 95% agresti-coull confidence interval. for unsupervised evaluation, we evaluated text embedding models and the bm25 algorithm to understand how they are able to handle pairs in different languages or even crosslingual pairs. factchecks are represented with their claims only. posts are represented with their main texts concatenated with the ocr transcripts. we use either the original texts or their english translations, depending on the version of the dataset that is reported. text embedding models (tems). we use various neural tems (reimers and gurevych, 2019) that encode texts into a vector space. these are usually based on pre-trained transformer language models fine-tuned as siamese networks to generate well-performing text embeddings. we use these models to embed both social media posts and factchecked claims into a common vector space. the retrieval is then reduced to calculating and sorting distances between vectors. ## supervised training to validate that our dataset can be used as a training set, we fine-tuned tems and evaluated their performance. we split the posts randomly into 80:10:10% train, development, and test sets. we used cosine or contrastive training losses to fine-tune the models. in both cases, both positive and negative pairs are required for training. we used our data as positive samples and random pairs as negative samples. we performed a hyperparameter search with gtr-t5 and mpnet tems (see section d). here, we report the best performing fine-tuned model we were able to achieve for both tems. the overall results for the test set are reported in we also observed that the tems were able to saturate the training set quite quickly, achieving 99.5%+ average precision after only a few epochs. this shows that our naive random selection of negative samples was too easy. the model can learn only a limited amount of information from such samples, and we would need a more elaborate scheme for generating more challenging negative samples. this could lead to further performance improvements. ## post-hoc results analysis the pairs, we obtained from the fact-checks, are only a subset of all the potentially valid pairs. this incompleteness limits our understanding of the dataset and also our evaluation. we decided to manually annotate a subset of the results generated by the methods to better understand what is missing from our data. we generated the top 10 fact-checks for the 87 test set posts that we knew had valid factchecks (see section 3). we used the 4 unsupervised and 2 supervised methods from section 5. these methods generated 3,390 unique pair predictions for these 87 posts. three authors went through each prediction and marked, whether they agreed with it, i.e., whether they found the factcheck to be valid and useful for the post. the agreement rates between the annotators were sufficiently high: 82.2%, 85.5% and 92.9%. we consider pairs where at least two annotators agreed to be correct. in total, the methods were able to find 719 correct pairs. 96 of these were present in our original dataset. this suggests that there is roughly 7× more pairs in our dataset than we had previously identified. no method was able to find 9 fact-checks out of 105 that were already in our dataset. of the 719 correct pairs, only 247 were monolingual, 136 were crosslingual with an english fact-check, and 336 were crosslingual with a non-english fact-check. the last category in particular is almost completely missing from our dataset. in ## discussion complexity of crosslingual evaluation. phenomena such as same language bias or false positive rate make the evaluation of multilingual and crosslingual datasets inherently complex. if we were to abstract the whole evaluation into a sin-gle number, as is often done in practice, we would have completely missed these pitfalls. without an in-depth evaluation, we might have been misled while applying our methods in practice, e.g., while developing helpful tools for fact-checkers. our evaluation procedures were previously impossible to develop in the absence of linguistically diverse pfcr datasets. ## conclusions in this paper, we introduced a new multilingual previously fact-checked claim retrieval dataset. our collection process yielded a unique and diverse dataset with a relatively small amount of noise in it. we believe that the evaluation of various methods is also insightful and can lead to the development of better fact-checking tools in the future. we believe that our dataset opens up many interesting research directions. we have, for example, barely scraped the surface of crosslingual learning in this work. applying various transfer learning methods (especially for low-resource languages) is an important future direction. 9 limitations 9.1 dataset noise. based on our annotation (see section 3), we expect that ∼ 13% of posts in our dataset do not contain the claim in the textual form. these are the cases when the claim being made on the social media is based on visual information. note, that the methods might still be able to retrieve correct factchecks for some of these posts, based on spurious correlations, e.g., overlaps in named entities.