Claim Matching Beyond E nglish to Scale Global Fact-Checking (2021)

Abstract: Manual fact-checking does not scale well to serve the needs of the internet. This issue is further compounded in non-English contexts. In this paper, we discuss claim matching as a possible solution to scale fact-checking. We define claim matching as the task of identifying pairs of textual messages containing claims that can be served with one fact-check. We construct a novel dataset of WhatsApp tipline and public group messages alongside fact-checked claims that are first annotated for containing “claim-like statements” and then matched with potentially similar items and annotated for claim matching. Our dataset contains content in high-resource (English, Hindi) and lower-resource (Bengali, Malayalam, Tamil) languages. We train our own embedding model using knowledge distillation and a high-quality “teacher” model in order to address the imbalance in embedding quality between the low- and high-resource languages in our dataset. We provide evaluations on the performance of our solution and compare with baselines and existing state-of-the-art multilingual embedding models, namely LASER and LaBSE. We demonstrate that our performance exceeds LASER and LaBSE in all settings. We release our annotated datasets, codebooks, and trained embedding model to allow for further research.

## introduction human fact-checking is high-quality but timeconsuming. given the effort that goes into factchecking a piece of content, it is desirable that a fact-check be easily matched with any content to which it applies. it is also necessary for factcheckers to prioritize content for fact-checking since there is not enough time to fact-check everything. in practice, there are many factors that affect whether a message is 'fact-check worthy' @xcite , but one important factor is prevalence. fact-checkers often want to check claims that currently have high viewership and avoid fact-checking 'fringe' claims as a fact-check could bring more attention to the claims-an understudied process known as amplification @xcite . while the number of exact duplicates and shares of a message can be used as a proxy for popularity, discovering and grouping together multiple messages making the same claims in different ways can give a more accurate view of prevalence. such algorithms are also important for serving relevant fact-checks via 'misinformation tiplines' on whatsapp and other platforms @xcite @xcite . identifying pairs of textual messages containing claims that can be served with one fact-check is a potential solution to these issues. the ability to group claim-matched textual content in different languages would enable fact-checking organizations around the globe to prioritize and scale up their efforts to combat misinformation. in this paper, we make the following contributions: (i) we develop the task of claim matching, (ii) we train and release an indian language xlm-r (i-xlm-r) sentence embedding model, (iii) we develop a multilingual annotated dataset across high-and lower-resource languages for evaluation, and (iv) we evaluate the ability of state-of-the-art sentence embedding models to perform claim matching at scale. we formally evaluate our methods within language but also show clusters found using our multilingual embedding model often have messages in different languages presenting the same claims. we release two annotated datasets and our codebooks to enable further research. the first dataset 1 shows examples of annotated pairs of messages from the second dataset. ## data sources the data used in this paper comes from a variety of sources. we use a mixture of social media (e.g., whatsapp) content alongside fact-checked claims, since it is essential for any claim-matching solution to be able to match content both among fact-checked claims and social media posts as well as within social media posts. among the prevalent topics in our data sources are the covid-19 pandemic, elections, and politics. tiplines. meedan, a technology non-profit, has been assisting fact-checking organizations to setup and run misinformation tiplines on whatsapp using their open-source software, check. a tipline is a dedicated service to which 'tips' can be submitted by users. on whatsapp, tiplines are phone numbers to which whatsapp users can forward potential misinformation to check for existing factchecks or request a new fact-check. the first tipline in our dataset ran during the 2019 indian elections and received 37,823 unique text messages. several additional always-on tiplines launched in december 2019 and ran throughout the 2020 calendar year. we obtained a list of the text of messages and the times at which they were submitted to these tiplines for march to may 2019 (indian election tipline) and for february 2020 to august 2020 (all other tiplines). we have no information beyond the text of messages and the times at which they were submitted. in particular, we have no information about the submitting users. whatsapp public groups. in addition to the messages submitted to these tiplines, we have data from a large number "public" whatsapp groups collected by garimella and eckles (2020) during the same time period as the indian election tipline. the dataset was collected by monitoring over 5,000 public whatsapp groups discussing politics in india, totaling over 2 million unique posts. for more information on the dataset, please refer to @xcite . such public whatsapp groups, particularly those discussing politics have been shown to be widely used in @xcite . fact-check reports. we aggregate roughly 150,000 fact-checks from a mixture of primary fact-checkers and fact-check aggregators. we employ aggregators such as google fact-check explorer, 3 gesis @xcite , and data commons, and include roughly a dozen factchecking organizations certified by the international fact-checking network with either global or geographically-relevant scope in our dataset. all fact-checks included at minimum a headline and a publish date, but typically also include a lead or the full text of the fact-check, as well as adjudication of the claim (e.g., truth or falsity), and sometimes include information of lesser value for our work such as author, categorization tags, or references to original content that necessitated the fact-check. ## data sampling & annotation to construct a dataset for claim matching, we design a two-step sampling and annotation process. we first sample a subset of items with potential matches from all sources and then annotate and select the ones containing "claim-like statements." in a second task, we annotate pairs of messages for claim similarity. one of the messages in each pair must have been annotated as containing a "claim-like statement" in the first annotation task. we sample possible matches in several ways in order to not unnecessarily waste annotator time. we describe these sampling strategies and other details of the process in the remainder of this section. ## results for some applications, it is good enough to be able to rank the most similar claims and treat the problem of claim matching as an information retrieval problem. this is the case, for example, when fact-checkers are examining possible matches to determine if a new content item matches a previous fact-check. we discuss the performance of information retrieval approaches in section 6.1. in many other applications, however, we seek a system that can determine if the claims in two items match without human intervention. these applications demand a classification approach: i.e., to determine whether two items match. this allows similar items to be grouped and fact-checkers to identify the largest groups of items with claims that have not been fact-checked. we discuss the performance of simple classification approaches in section 6.2. ## discussion & conclusions scaling human-led fact-checking efforts requires matching messages with the same claims. in this paper, we train a new model and create an evaluation dataset that moves beyond english and american politics. our system is being used in practice to support fact-checking organizations. we find that the embedding models can generally match messages with the same claims. performance for matching fact-checks slightly exceeds that for matching social media items. this makes sense, given that fact-checks are written by professional journalists and generally exhibit less orthographical variation than social media items. too few examples of fact-checks correctly matched a social media item to evaluate performance in that setting. this is not a major limitation since nearly every fact-check starts from a social media item. so, in practice we only need to be able to match social media items to one another in order to locate other social media items having the same claims as the item that led to a fact-check. we evaluate claim matching within each language, but the embedding models are all multilingual and could serve to match claims across languages. bm25 is not multilingual, but elasticsearch can index embeddings directly. previously de britto almeida and santos (2020) developed a elasticsearch plugin to query embeddings by cosine distance, but since version 7.3 of elasticsearch this functionality is now available natively in elasticsearch @xcite , meaning a large set of embeddings can be searched efficiently to find near matches across languages. as a proof of concept, we took the 37,823 unique text messages sent to the indian election tipline and clustered them using i-xlm-r and online, singlelink hierarchical clustering with a threshold of 0.90. we found 1,305 clusters with 2 or more items; the largest cluster had 213 items. we hired an indian journalist with experience fact-checking during the indian 2019 elections to annotate each of the 559 clusters with five or more items by hand. the annotation interface presented three examples from each cluster: one with the lowest average distance to all other messages in the cluster, one with the highest distance, and one message chosen randomly. in 137 cases the examples shown for annotation were from multiple languages, and in 132 of those cases the journalist was able to identify the same claims across multiple languages. although preliminary, this demonstrates the feasibility and importance of multilingual claim matching with these methodsan area we hope further work will tackle. our findings are supporting over 12 factchecking organizations running misinformation tiplines. the deployed system uses i-xlm-r and automatically groups text messages with similarities over 0.95 and recommends possible matches from less-similar candidates that fact-checking organizations can confirm or reject. matches can also be added manually. initial feedback from the factcheckers has been positive, and we are collecting data for further research and evaluation. we prioritized the well-being of annotators and the privacy of whatsapp users throughout this research. our data release conforms to the fair principles @xcite . we have no identifying information about whatsapp users and any references to personally identifiable information in messages such as phone numbers, emails, addresses and license plate numbers are removed to preserve user privacy. we worked closely with our annotators preparing them for the risk of hateful content, encouraging frequent breaks, and paying well-above minimum wage. we took a compassionate response to covid disruptions and other life stresses even when this meant less annotated data than was originally envisioned.