Fighting the COVID -19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society (2021)

Abstract: With the emergence of the COVID-19 pandemic, the political and the medical aspects of disinformation merged as the problem got elevated to a whole new level to become the first global infodemic. Fighting this infodemic has been declared one of the most important focus areas of the World Health Organization, with dangers ranging from promoting fake cures, rumors, and conspiracy theories to spreading xenophobia and panic. Addressing the issue requires solving a number of challenging problems such as identifying messages containing claims, determining their check-worthiness and factuality, and their potential to do harm as well as the nature of that harm, to mention just a few. To address this gap, we release a large dataset of 16K manually annotated tweets for fine-grained disinformation analysis that (i) focuses on COVID-19, (ii) combines the perspectives and the interests of journalists, fact-checkers, social media platforms, policy makers, and society, and (iii) covers Arabic, Bulgarian, Dutch, and English. Finally, we show strong evaluation results using pretrained Transformers, thus confirming the practical utility of the dataset in monolingual vs. multilingual, and single task vs. multitask settings.

## introduction the rise of social media has made them one of the main channels for information dissemination and consumption. as a result, nowadays, many people rely on social media as their primary source of news @xcite , attracted by the broader choice of information sources and by the ease for anybody to become a news producer. unfortunately, the democratic nature of social media has raised questions about the quality and the factuality of the information that is shared on these platforms. eventually, social media have become one of the main channels to spread disinformation. figure 1 demonstrates how online users discuss topics related to covid-19 in social media. we can see that the problem goes beyond factuality: there are tweets spreading rumors (figure 1a ), instilling panic (figure 1b ), making jokes (figure 1c ), promoting fake cures (figure 1d ), spreading xenophobia, racism, and prejudices (figure 1e ), or promoting conspiracy theories (figure 1h ). other examples in figure 1 contain information that could be potentially useful and might deserve the attention of government entities. for example, the tweet in figure 1f blames the authorities for their inaction regarding covid-19 testing. the tweet in figure 1g is useful both for policy makers and for the general public as it discusses action taken and suggest actions that probably should be taken elsewhere to fight the pandemic. for the tweets in figure 1 , it is necessary to understand whether the information is correct, harmful, calling for action to be taken by relevant authorities, etc. rapidly sorting these questions is crucial to help organizations channel their efforts, and to counter the spread of disinformation, which may cause panic, mistrust, and other problems. addressing these issues requires significant effort in terms of (i) defining comprehensive annotation guidelines, (ii) collecting tweets about covid-19 and sampling from them, (iii) annotating the tweets, and (iv) training and evaluating models. given the interconnected nature of these issues, it is more efficient to address them simultaneously. with this consideration in mind, we adopt a multifaceted approach, which is motivated by engaging with different stakeholders such as journalists and policy makers. we focused on three key aspects, which are formulated into seven questions: (i) check worthiness and veracity of the tweet (q1-4 and q5). (ii) harmfulness to society (q6); and (iii) call for action addressing a government / policy makers (q7). q1-q5 were motivated by conversations with journalists and professional factcheckers, while q6-q7 were formulated in conversations with a ministry of public health. our contributions can be summarized as follows: a number of datasets were created as part of shared tasks. in most cases, they performed their own annotation, either @xcite manually, e.g., the se-meval tasks on determining the veracity of rumors @xcite , propaganda detection in news articles and memes @xcite , fact-checking in community question answering forums @xcite , the clef checkthat! lab on identification and verification of claims @xcite @xcite @xcite , or (b) using crowdsourcing, e.g., the fever task on fact extraction and verification, focusing on claims about wikipedia content @xcite . unlike our work, the above datasets did not focus on tweets (they used claims from news, speeches, political debates, community question answering fora, or were just made up by human annotators; rumoureval is a notable exception), targeted factuality only (we cover a number of other issues), were limited to a single language (typically english; except for clef), and did not focus on covid-19. check-worthiness estimation another relevant research line is on detecting check-worthy claims in political debates using manual annotations @xcite or by observing the selection of fact-checkers @xcite @xcite . ## experimental setup we experimented with binary and multiclass settings for all languages, using deep contextualized text representations based on large-scale pretrained transformer models such as bert, mbert, roberta, xlm-r, etc. we further performed multitask and cross-language learning, and we modeled the social context of the tweet, as well as the propagandistic nature of the language used. ## advanced experiments next, we performed some additional, more advanced experiments, including multilingual training, modeling the twitter context, the use of propagandistic language, and whether the user is likely to be a bot, as well as multitask learning. we describe each of these experiments in more detail below. ## conclusion and future work we presented a large manually annotated dataset of covid-19 tweets, aiming to help in the fight against the covid-19 infodemic. the dataset combines the perspectives and the interests of journalists, fact-checkers, social media platforms, policymakers, and society as a whole. it includes tweets in arabic, bulgarian, dutch, and english, and we are making it freely available to the research community. we further reported a number of evaluation results for all languages using various transformer architectures. moreover, we performed advanced experiments, including multilingual training, modeling the twitter context, the use of propagandistic language, and whether the user is likely to be a bot, as well as multitask learning. in future work, we plan to explore multimodality and explainability @xcite . we further want to model the task as a multitask ordinal regression @xcite , as q2-q5 are defined on an ordinal scale. moreover, we would like to put the data and the system in some practical use; in fact, we have already used them to analyze disinformation about covid-19 in bulgaria @xcite and qatar @xcite . finally, the data will be used in a shared task at the clef-2022 checkthat! lab; part of it was used for the nlp4if-2021 shared task @xcite .