# GCDH at WNUT -2020 Task 2: BERT -Based Models for the Detection of Informativeness in E nglish COVID -19 Related Tweets (2020)

Abstract: In this system paper, we present a transformer-based approach to the detection of informativeness in English tweets on the topic of the current COVID-19 pandemic. Our models distinguish informative tweets, i.e. tweets containing statistics on recovery, suspected and confirmed cases and COVID-19 related deaths, from uninformative tweets. We present two transformer-based approaches as well as a Naive Bayes classifier and a support vector machine as baseline systems. The transformer models outperform the baselines by more than 0.1 in F1-score, with F1-scores of 0.9091 and 0.9036. Our models were submitted to the shared task Identification of informative COVID-19 English tweets WNUT-2020 Task 2.

## introduction as of the end of august 2020, the outbreak of the covid-19 pandemic has lead to 24.5 million cases world-wide and affected individuals, communities and nations in all parts of the world 1 . in order to receive reliable data on cases in their area, users rely on information systems, such as the coronavirus resource center introduced by the johns hopkins university. 2 however, those systems mainly utilize statistics provided by official health institutes, such as the world health organisation or the robert koch institute. in order to confirm the numbers of covid-19 related confirmed cases, recoveries or deaths, those official information channels release data once a day. alternative sources, e.g. social media outlets such as twitter, publish a vast amount of data in real-time. therefore, the wnut-2020 shared task 2 @xcite called for systems that can distinguish informative twitter messages, so-called tweets, from uninformative data. a sophisticated informativeness detection system can be used to update covid-19 information systems for users world-wide on a real-time basis. in this paper, we present different machinelearning approaches on the identification of informative tweets. we first evaluate naive bayes and support-vector-machine approaches with handcrafted features. subsequently, we employ transformer-based models, such as covid-twitter-bert @xcite , which provide the best results. this paper is structured as follows: section 2 explains the task and the data provided by the hosts of the shared task, section 3 explains different approaches on the machine-learning systems, sections 4 and 5 present and discuss results. we finish this paper with a conclusion in section 6 and an outview on future work in section 7. our source code is available at https://gitlab. gwdg.de/tillmann.doenicke/wnut-2020 . ## data the organizers of the shared task provided raw data of a tweet id, tweet text and the label "informative" or "uninformative". participants were free to make use of additional data @xcite . we used the suggested split of the dataset with 7,000 tweets for training and 1,000 for validation. ## results bert-based models perform better than baseline models. the best result was achieved using the ct-bert model trained on the domain-specific twitter data on the topic of covid-19 ("ct-bert-1" in ## discussion due to the fact that the transformer-based models were pre-trained on much larger, external datasets @xcite , it is unsurprising that they outperform our baseline models, which were solely trained on the shared task's dataset. our transformer-based models predict informative tweets slightly better than uninformative tweets. on the validation set, the bert model delivers 115 falsely classified tweets, 28 of which were wrongly classified as uninformative and 87 which were classified as informative. the ct-bert-1 model falsely predicted 25 tweets as uninformative and 52 as informative. therefore, the main difference lies in the prediction of uninformative tweets, for which ct-bert-1 yields better results. this leads to a better recall of the ct-bert-1 model compared to the bert model. upon manual inspections, the falsely classified tweets fall into different categories, such as tweets that show statistics but also a personal opinion, personal stories and events, or political statements and protective measures against covid (e.g. "amazon and facebook ask seattle employees to work from home after coronavirus cases httpurl" annotated as informative). we cannot determine a pattern for the cases in which ct-bert-1 yields better predictions than bert. without the guidelines on how tweets were annotated for the dataset, it is hard to assess why certain tweets were initially labelled as informative or uninformative in the data provided. therefore, we were not able to identify the reason why certain predictions were incorrect. ## conclusion we show in our paper how a transformer-based approach can yield good results with relatively little fine-tuning. our best-performing model, the ct-bert-1 model, predicted informativeness with an f1-score of 0.9091 in the final w-nut task 2 evaluation. detecting informativeness of a small text unit, such as a tweet, is one of the many steps that are required in order to build trustworthy real-time news applications.