UP enn HLP at WNUT -2020 Task 2 : Transformer models for classification of COVID 19 posts on T witter (2020)

Abstract: Increasing usage of social media presents new non-traditional avenues for monitoring disease outbreaks, virus transmissions and disease progressions through user posts describing test results or disease symptoms. However, the discussions on the topic of infectious diseases that are informative in nature also span various topics such as news, politics and humor which makes the data mining challenging. We present a system to identify tweets about the COVID19 disease outbreak that are deemed to be informative on Twitter for use in downstream applications. The system scored a F1-score of 0.8941, Precision of 0.9028, Recall of 0.8856 and Accuracy of 0.9010. In the shared task organized as part of the 6th Workshop of Noisy User-generated Text (WNUT), the system was ranked 18th by F1-score and 13th by Accuracy.

## introduction the covid19 pandemic caused by the coronavirus (ncov) has presented a unique challenge to the public health research community in the areas of tracking localized and community level transmissions for enforcing effective mobility restrictions and interventions to curb further virus spread. while traditional sources of infection numbers and fatalities include testing facilities and healthcare providers, many new non-traditional sources of information such as social media, wastewater analysis and mobility statistics that may serve as biomarkers for presence of infections in the community. in this work, we focus on using natural language processing (nlp) for mining social media posts for tweets that mention that can be used for public health monitoring purposes or dissemination of information. we accomplish this by introducing a classifier that can be used as a component of an information processing pipeline to detect informative tweets from posts that mention keywords related to discussions around coronavirus. the system presented in this work was developed as part of the w-nut 2020 shared task 2 @xcite where the objective of the task was to classify a given post as informative or uninformative. the rest of the document is structured as follows: we briefly discuss previous related work on covid19 surveillance on twitter in the background section. we describe the annotated dataset and system implementation in the materials and methods section followed by preliminary and final evaluation results in the results section. finally, we discuss error analysis, limitations and future directions in the discussion section. ## background researchers working on noisy user texts such as posts on social media mining have proposed various methods and systems for monitoring infectious disease transmissions and natural disasters the classification categories themselves have varied from sentiments expressed in posts @xcite and relatedness to the disease @xcite to misinformation detection @xcite and personal reports of exposures or test results @xcite . each dataset contains tweets and annotations that can be processed by information processing pipelines using nlp and machine learning based classification techniques for possible applications in public health using epidemiological analysis. ## materials and methods the dataset annotated for the task consists of 10,000 tweets that mentioned terms related to covid19. this included a total of 4719 tweets labeled as informative and 5281 tweets labeled as uninformative. each tweet was annotated by 3 independent annotators with an inter-annotator agreement score of fleiss' kappa of 0.818. the dataset was split into training set (70%), validation set (10%) and test set (20%) for the purposes of development and evaluation of the classification model. we tabulate further details of the splits in ## results the final detailed results of the task on the validation and test sets are shown in ## discussion on closer analysis of initial errors on the validation set we found that many tweets were difficult to determine as being in either classes. on performing an 8-fold cross validation, we found that fold 1 which the default split of training and validation set had the lowest f1-scores at 0.92 whereas folds 2-8 had f1-scores in the ranges of 0.94-0.96. on closer analysis of fold 1, we found that many tweets may have been annotated incorrectly in the validation set which may have introduced errors in the final model and hence may require further analyses of the annotations for development of better models. overall, transformer models performed significantly better than non-transformer models that we trained on. however, we note that the nontransformer models could be improved by using word-embeddings and attention features in layers among feedforward and cnn architectures. the current system proposes a simple classification method which may be useful in removing tweets that are deemed uninformative for use in downstream epidemiological analyses. further research is required to assess the utility of the tweets obtained from such collections of informative tweets for disease tracking and analyses of symptoms. ## conclusion social media mining offers a non-traditional avenue to extract epidemiological and population level statistics from discussions around a topic. such noisy information domains presents an nlp challenge for extracting meaningful information for use in downstream applications. in this work, we present a system to identify informative tweets on english language twitter posts on the topic of covid19 pandemic for use in downstream tasks such as public health monitoring and epidemiological studies. we use a classification approach to identifying such tweets and the final system scored a f1-score of 0.8941, precision of 0.9028, recall of 0.8856 and accuracy of 0.9010. in the shared task organized as part of the 6th workshop of noisy user-generated text (wnut), the system was ranked 18th by f1-score and 13th by accuracy.