X -Fact: A New Benchmark Dataset for Multilingual Fact Checking (2021)

Abstract: In this work, we introduce : the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata and evidence from news stories retrieved using a search engine. Empirically, our best model attains an F-score of around 40%, suggesting that our dataset is a challenging benchmark for the evaluation of multilingual fact-checking models.

## introduction curbing the spread of fake news and misinformation on the web has become an important societal challenge. several fact-checking initiatives, such as politifact, 1 expend a significant amount of manual labor to investigate and determine the truthfulness of viral statements made by public figures, organizations, and social media users. of course, since this process is time-consuming, often, a large number of falsified statements go unchecked. with the aim of assisting fact-checkers, researchers in nlp have sought to develop computational approaches to fact-checking @xcite @xcite . many such works use the fever dataset, which contains claims extracted from wikipedia documents @xcite . using real-world claims, wang (2017) introduced liar, a dataset 1 https://www.politifact.com/ with 12,836 claims from politifact. recently, @xcite introduced multifc, an even larger corpus of 34,918 claims collected from 26 fact-checking websites. although misinformation transcends countries and languages @xcite , much of the recent work focuses on claims and statements made in english. developing automated fact checking (afc) systems in other languages is much more challenging, the primary reason being the absence of a manually annotated benchmark dataset for those languages. moreover, there are fewer fact-checkers in these languages, and as a result, a non-english monolingual dataset will inevitably be small and less effective in developing fact-checking systems. as recent research points out, a possible solution in dealing with data scarcity is to train multilingual models @xcite @xcite . indeed, this finding motivates us to construct a large multilingual resource that the research community can use to further the development of fact-checking systems in languages other than english. recent efforts in the construction of a multilingual dataset are limited, both in scope and in size @xcite . for instance, fakecovid, a dataset introduced by shahi and nandini (2020) contains 3066 non-english claims about covid-19. in comparison, x-fact contains 31,189 general domain non-english claims from 25 languages. moreover, fakecovid contains only two labels, namely, false, and others. we argue that this is undesirable, as fact checking is a fine-grained classification task. due to subtle differences in language, most claims are neither entirely true nor entirely false @xcite . in contrast, our dataset contains seven labels-we make distinctions between true, mostly true, half-true etc. examples from german and brazilian portuguese. in summary, our contributions are: the x-fact dataset, and the code for our experiments, can be obtained at https://github.com/ utahnlp/x-fact . ## conclusion we presented x-fact, the currently largest multilingual dataset for fact-checking. compared to the prior work, x-fact is an order of magnitude larger, enabling the exploration of large transformerbased multilingual approaches to fact-checking. we presented results for several multilingual modeling methods and showed that the models find this new dataset challenging. we envision our dataset as an important benchmark in development and evaluation of multilingual approaches to factchecking. because of these reasons, a large number of websites in a number of languages could not be crawled. there are two ways we obtain our claims, labels, and other metadata. one is the google's fact check explorer (gfce) 11 , and the other is by crawling from the respective fact-checking website. in case, the links are available on gfce, we download other metadata by visiting the website. also, we will release the label mapping we created along with the dataset. appendix a provides more details on the dataset we collected.