FANG - COVID : A New Large-Scale Benchmark Dataset for Fake News Detection in G erman (2021)

Abstract: As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an ‘infodemic’ – a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society. To combat this infodemic, there is an urgent need for benchmark datasets that can help researchers develop and evaluate models geared towards automatic detection of disinformation. While there are increasing efforts to create adequate, open-source benchmark datasets for English, comparable resources are virtually unavailable for German, leaving research for the German language lagging significantly behind. In this paper, we introduce the new benchmark dataset FANG-COVID consisting of 28,056 real and 13,186 fake German news articles related to the COVID-19 pandemic as well as data on their propagation on Twitter. Furthermore, we propose an explainable textual- and social context-based model for fake news detection, compare its performance to “black-box” models and perform feature ablation to assess the relative importance of human-interpretable features in distinguishing fake news from authentic news.

## introduction the online availability and rapid dissemination of 'disinformation' and 'fake news' -cover terms for various types of false, inaccurate, or misleading information, typically related to emerging and timesensitive events -have become a global challenge over the past 10 years @xcite . never has the scale of this challenge been clearer than in the corona crisis: as the world continues to fight the coronavirus disease 2019 (covid-19) pandemic, it is simultaneously fighting an 'infodemic' -a flood of disinformation and spread of conspiracy theories leading to the division of society @xcite . in order to combat the spread and consumption of disinformation, it is of essential importance to develop diagnostic and predictive analysis models that can be used to understand how and why disinformation is created and spread as well as to uncover hidden and unexpected aspects of disinformation content. a key aspect that has a significant impact on the detection of disinformation is the existence of high-quality benchmark datasets. while there are increasing efforts to create adequate, opensource benchmark datasets for english, comparable resources are virtually unavailable for german (but see @xcite ). accordingly, research on disinformation detection for german is clearly lagging behind. in order to fill the research gap, the main goal of this paper is to introduce fang-covid foot_0 , an entire-article covid-19 benchmark dataset for the german language, designed based on existing english language benchmark datasets and labeled using distant supervision. furthermore, we present the results of benchmark experiments on the dataset using explainable textual-and social context-based models for fake news detection and compare their performance to those of classifiers based on a fine-tuned bert language model. finally, we conduct feature ablation experiments to assess the relative importance of human-interpretable features in distinguishing fake news from authentic news. this paper is structured as follows: in section 2, we provide a concise overview of existing datasets and approaches for fake news detection, focusing on work related to covid-19 and the german language. in section 3, we introduce the fang-covid dataset. in section 4, we propose an explainable fake news detection system, compare its performance to a black-box model and introduce the ablation algorithm used to analyze the importance of certain features, before summarizing the results and our contribution in section 5. ## related work in recent years, with the growing recognition of the importance of understanding and detecting fake news, increasing efforts have been made to combat disinformation, resulting in the creation of largescale annotated datasets for fake news detection for the english language @xcite @xcite . for the german language, first steps towards creating suitable datasets have been made @xcite . to date, however, existing datasets are too small to enable the development of state-of-the-art deep learning models for fake news detection. current approaches to automatic detection of english fake news @xcite @xcite have taken into account both textual news content and also social context based on social media data (for recent reviews of disinformation detection, see @xcite , @xcite , @xcite and @xcite ). language-based approaches have an advantage over knowledge-based or propagationbased approaches foot_1 , because (1) they enable near real-time feedback (proactive rather than retroactive), i.e. they are not restricted to being applied only a posteriori @xcite and (2) they are scalable. in the remainder of this section, we provide a concise overview of the existing covid-19 fake news datasets and approaches to detect disinformation regarding the virus. ## introducing the fang-covid dataset in this section, we provide a detailed description of the methodology used to compile the fang-covid dataset and present its composition. the collection and labelling of news articles was based on news publisher bias lists compiled by media professionals, following the approach taken in @xcite . for gathering true news articles, we relied on three established, reputable mainstream newspapers whose quality is recognized by media experts @xcite . for the collection of fake news articles, we utilized independent fact-checking organizations such as correctiv foot_2 and newsguard foot_3 . the latter published a list in january 2021 containing over 30 german news sources that are classified as unreliable news publishers based on criteria such as the frequency of false information being posted, the responsible presentation of information and sources and the clear separation of opinion and fact foot_4 . based on this data and further research on its listed sources via trustworthy newspapers such as frankfurter allgemeine zeitung, spiegel or zeit, we selected ten publishers whose articles we would label as 'fake'. an overview of the distributions of news articles and tweets across publishers is presented in for every one of our news sources, we scraped the header, date of publication and textual content of articles that were published between february 2020 to mid march 2021 and contained one of the following covid-related keywords: corona, covid, infektion, lockdown, impfen, impfung, impfstoff. the selection of news based on the keywords ensured that all articles are related to events surrounding the coronavirus pandemic. the collected textual data were cleaned and preprocessed by transforming dates of publication from handwritten formats to the uniform datetime format and the removal of html tags. a schematic representation of the procedure used to compile fang-covid is shown in figure 1 . in total, the fang-covid dataset comprises 41,242 news articles: 28,056 articles from three publishers are labeled as 'real' and 13,186 articles are labeled as 'fake' based on the reliability of their source (see tables 2 and 3 for details on the composition of the dataset). on average, news articles in fang-covid contain 803 word tokens and 43.8 sentences, with similar distributions for fake and real news articles. with regards to political orientation, the majority of the fake news publishers exhibited tendencies towards right wing populism, as common topics by these publishers involve allegations against immigrants (a topic that has been especially prevalent in germany since the refugee crisis in 2015). the articles from unreliable sources are also characterized by a higher proportion of articles favouring the right-wing populist party "alternative für deutschland" (afd) (see figure 4 in the appendix). to facilitate research on the diffusion of fake news in social networks, fang-covid also contains rich information relating to the articles' spreading on social media. these data were collected using the snscrape foot_5 library, a python scraper for social networking services. we gathered all tweets referring to any of the articles in the dataset, as well as reactions to these in the form of likes or retweets and additional information, such as which device the tweet was posted from as well as userrelated data, such as the follower count, the number of posts or the date a user joined twitter. each news article is associated with a json array containing all tweets sharing its url, ordered by the time they were posted. each tweet is represented as a json object containing specific information about each tweet, such as the number of likes and retweets as well as extensive data about the user sharing each tweet such as their follower count, their number of posts since joining twitter as well as visual information in the form of their profile picture and banner. overall, fang-covid contains social media information related to the news articles from a total of 363,393 tweets. on average, a real news article from our dataset was shared 8.4 times while a fake news article was shared 9.7 times. out of all users sharing articles from our dataset, 52,289 distinct user profiles reposted real news articles while 7,861 users shared news articles labeled as fake. ## fang-covid: benchmark experiments in this section, we first describe the textual and social context representations used in the fake news detection model and the experimental setup, before presenting the empirical results of classification models trained on interpretable features and contextualized word embeddings. specifically, in order to establish benchmarks for fake news detection on the fang-covid dataset and to provide initial insights into the features and approaches that prove to be effective for the detection of german fake news regarding covid-19, we evaluate detection models based on interpretable term frequency inverse document frequency (tf-idf) and linguistic complexity features and compare their performance to classifiers based on a fine-tuned bert language model with and without social context information. ## conclusion the development of effective diagnostic and predictive models needed to combat the spread and consumption of disinformation requires high-quality benchmark datasets. in this paper, we introduced the fang-covid dataset, a new dataset for fake news detection in german, consisting of 40k+ real and fake news articles related to the covid-19 pandemic along with data on their propagation on twitter. to facilitate fake news related research, the dataset is publicly available to the research community on github. we further conducted benchmark experiments on the data that incorporated both uninterpretable and interpretable features to language-based fake news detection in conjunction with social media engagement and profile data. the results of these experiments indicate that models that combine interpretable language features with social network data can achieve comparable detection performance relative to those based on contextualized word embeddings. this finding is important in the light of the growing recognition of the need to move away from pure black-box models towards interpretable models for solving practical problems, in particular in the context of critical industries, including healthcare, criminal justice, and news @xcite . this is due to the fact that human experts in a given application domain need both accurate but also understandable models @xcite . potentially, our work can be expanded upon with the addition of further linguistic features that can increase the predictive power of an interpretable model to match that of black box models. furthermore, a wide range of data from the social context information provided in fang-covid can be integrated to build more sophisticated fake news detection models and to study the propagation of german fake news on social media. to address issues associated with the high correlation between the source of a news article and its class, approaches integrating adversarial objectives @xcite could be evaluated.