WNUT -2020 Task 2: Identification of Informative COVID -19 E nglish Tweets (2020)

Abstract: In this paper, we provide an overview of the WNUT-2020 shared task on the identification of informative COVID-19 English Tweets. We describe how we construct a corpus of 10K Tweets and organize the development and evaluation phases for this task. In addition, we also present a brief summary of results obtained from the final system evaluation submissions of 55 teams, finding that (i) many systems obtain very high performance, up to 0.91 F1 score, (ii) the majority of the submissions achieve substantially higher results than the baseline fastText (Joulin et al., 2017), and (iii) fine-tuning pre-trained language models on relevant language data followed by supervised training performs well in this task.

## introduction as of late-september 2020, the covid-19 coronavirus pandemic has led to about 1m deaths and 33m infected patients from 213 countries and territories, creating fear and panic for people all around the world. 1 recently, much attention has been paid to building monitoring systems (e.g. the johns hopkins coronavirus dashboard) to track the development of the pandemic and to provide users the information related to the virus, 2 e.g. any new suspicious/confirmed cases near/in the users' regions. it is worth noting that most of the "official" sources used in the tracking tools are not frequently kept up to date with the current pandemic situation, e.g. who updates the pandemic information only once a day. those monitoring systems thus use social network data, e.g. from twit-ter, as a real-time alternative source for updating the pandemic information, generally by crowdsourcing or searching for related information manually. however, the pandemic has been spreading rapidly; we observe a massive amount of data on social networks, e.g. about 3.5m of covid-19 english tweets posted daily on the twitter platform @xcite in which the majority are uninformative. thus, it is important to be able to select the informative tweets (e.g. covid-19 tweets related to new cases or suspicious cases) for downstream applications. however, manual approaches to identify the informative tweets require significant human efforts, do not scale with rapid developments, and are costly. to help handle the problem, we propose a shared task which is to automatically identify whether a covid-19 english tweet is informative or not. our task is defined as a binary classification problem: given an english tweet related to covid-19, decide whether it should be classified as informative or uninformative. here, informative tweets provide information about suspected, confirmed, recovered and death cases as well as the location or travel history of the cases. the following example presents an informative tweet: informative update: uganda health minister jane ruth aceng has confirmed the first #coronavirus case in uganda. the patient is a 36-yearold ugandan male who arrived from dubai today aboard ethiopian airlines. patient travelled to dubai 4 days ago. #coron-aviruspandemic the goals of our shared task are: (i) to develop a language processing task that potentially impacts research and downstream applications, and (ii) to provide the research community with a new dataset for identifying informative covid-19 english tweets. to achieve the goals, we manually construct a dataset of 10k covid-19 english tweets with informative and unin-formative labels. we believe that the dataset and systems developed for our task will be beneficial for the development of covid-19 monitoring systems. all practical information, data download links and the final evaluation results can be found at the codalab website of our shared task: https://competitions.codalab. org/competitions/25845 . ## results in total, 121 teams spreading across 20 different countries registered to participate in our wnut-2020 task 2 during the system development phase. of those 121 teams, 55 teams uploaded their submissions for the final evaluation phase. 5 we report results obtained for each team in not surprisingly, ct-bert, resulted in by continuing pre-training from the pre-trained bertlarge model on a corpus of 22.5m covid-19 related tweets, is utilized in a large number of the highly-ranked systems. in particular, all of top 6 teams including nutcracker, nlp north, uit-hse @xcite , #gcdh @xcite , loner and phonemer @xcite utilize ct-bert. that is why we find slight differences in their obtained f 1 scores. in addition, ensemble techniques are also used in a large proportion (61%) of the participating teams. specifically, to obtain the best performance, the top 10 teams, except nlp north, #gcdh and loner, all employ ensemble techniques. ## conclusion in this paper, we have presented an overview of the wnut-2020 task 2 "identification of informative covid-19 english tweets": (i) provide details of the task, data preparation process, and the task organization, and (ii) report the results obtained by participating teams and outline their commonly adopted approaches. we receive registrations from 121 teams and final system evaluation submissions from 55 teams, in which 34/55 teams contribute detailed system descriptions. the evaluation results show that many systems obtain a very high performance of up to 0.91 f 1 score on the task, using pre-trained language models which are fine-tuned on unlabelled covid-19 related tweets (ct-bert) and are subsequently trained on this task.