Findings of the NLP 4 IF -2021 Shared Tasks on Fighting the COVID -19 Infodemic and Censorship Detection (2021)

Abstract: We present the results and the main findings of the NLP4IF-2021 shared tasks. Task 1 focused on fighting the COVID-19 infodemic in social media, and it was offered in Arabic, Bulgarian, and English. Given a tweet, it asked to predict whether that tweet contains a verifiable claim, and if so, whether it is likely to be false, is of general interest, is likely to be harmful, and is worthy of manual fact-checking; also, whether it is harmful to society, and whether it requires the attention of policy makers. Task 2 focused on censorship detection, and was offered in Chinese. A total of ten teams submitted systems for task 1, and one team participated in task 2; nine teams also submitted a system description paper. Here, we present the tasks, analyze the results, and discuss the system submissions and the methods they used. Most submissions achieved sizable improvements over several baselines, and the best systems used pre-trained Transformers and ensembles. The data, the scorers and the leaderboards for the tasks are available at http://gitlab.com/NLP4IF/nlp4if-2021 .

## introduction social media have become a major communication channel, enabling fast dissemination and consumption of information. a lot of this information is true and shared in good intention; however, some is false and potentially harmful. while the so-called "fake news" is not a new phenomenon, e.g., the term was coined five years ago, the covid-19 pandemic has given rise to the first global social media infodemic. the infodemic has elevated the problem to a whole new level, which goes beyond spreading fake news, rumors, and conspiracy theories, and extends to promoting fake cure, panic, racism, xenophobia, and mistrust in the authorities, among others. identifying such false and potentially malicious information in tweets is important to journalists, fact-checkers, policy makers, government entities, social media platforms, and society. a number of initiatives have been launched to fight this infodemic, e.g., by building and analyzing large collections of tweets, their content, source, propagators, and spread @xcite @xcite . yet, these efforts typically focus on a specific aspect, rather than studying the problem from a holistic perspective. here we aim to bridge this gap by introducing a task that asks to predict whether a tweet contains a verifiable claim, and if so, whether it is likely to be false, is of general interest, is likely to be harmful, and is worthy of manual fact-checking; also, whether it is harmful to society, and whether it requires the attention of policy makers. the task follows an annotation schema proposed in @xcite . while the covid-19 infodemic is characterized by insufficient attention paid to the problem, there are also examples of the opposite: tight control over information. in particular, freedom of expression in social media has been supercharged by a new and more effective form of digital authoritarianism. political censorship exists in many countries, whose governments attempt to conceal or to manipulate information to make sure their citizens are unable to read or to express views that are contrary to those of people in power. one such example is sina weibo, a chinese microblogging website with over 500 million monthly active users, which sets strict control over its content using a variety of strategies to target censorable posts, ranging from keyword list filtering to individual user monitoring: among all posts that are eventually censored, nearly 30% are removed within 5-30 minutes, and for 90% this is done within 24 hours @xcite . we hypothesize that the former is done automatically, while the latter involves human censors. thus, we propose a shared task that aims to study the potential for automatic sensorship, which asks participating systems to predict whether a sina weibo post will be censored. ## related work in this section, we discuss studies relevant to the covid-19 infodemic and to censorship detection. ## task organization in this section, we describe the overall task organization, phases, and evaluation measures. ## evaluation results for task 1 below, we describe the baselines, the evaluation results, and the best systems for each language. ## conclusion and future work we have presented the nlp4if-2021 shared tasks on fighting the covid-19 infodemic in social media (offered in arabic, bulgarian, and english) and on censorship detection (offered in chinese). in future work, we plan to extend the dataset to cover more examples, e.g., from more recent periods when the attention has shifted from covid-19 in general to vaccines. we further plan to develop similar datasets for other languages.