CIA _ NITT at WNUT -2020 Task 2: Classification of COVID -19 Tweets Using Pre-trained Language Models (2020)

Abstract: This paper presents our models for WNUT2020 shared task2. The shared task2 involves identification of COVID-19 related informative tweets. We treat this as binary text clas-sification problem and experiment with pre-trained language models. Our first model which is based on CT-BERT achieves F1-scoreof 88.7% and second model which is an ensemble of CT-BERT, RoBERTa and SVM achieves F1-score of 88.52%.

## introduction as of september 07,2020 covid-19 coronavirus infected 27.3m people and caused 887k deaths 1 . real time updates regarding the number of infected cases and death cases is given in dashboards. these dashboards make use of information from social networking sites like twitter. as majority of the tweets posted online are uninformative, it is necessary identify the informative tweets which include useful information related to recovered, suspected, confirmed and death cases as well as location or travel history of the cases. the wnut 2020 shared task2 involves identification of informative tweets. we treat this as binary text classification problem. prior to 2018, most of the text classification models are based on @xcite or recurrent neural network(rnn). these models are shallow in nature and cannot learn more informative features from the input. moreover as these models are to trained from scratch, they require more number of training instances @xcite . recently pre-trained language models like bert @xcite , roberta @xcite achieved significant improvements in many of the natural language processing tasks @xcite ). in this paper, we propose two models to identify informative covid-19 tweets. first model is based on covid-twitter-bert (ct-bert) which is a bert-large based model which is further trained on 160m corona virus related tweets @xcite . second model is ensemble of ct-bert, roberta and svm @xcite . as ct-bert is initialized from bertlarge weights and further pre-trained on covid tweets, it has two advantages compared to bertlarge which is pre-trained on generic less noisy texts. first advantage is, ct-bert includes domain as well as specific information and second advantage is, ct-bert can better handle noisy texts like tweets. our ct-bert based model achieves f1-score of 88.87% and ensemble model achieves f1-score of 88.52% 2 related work ## results to identify informative tweets related to corona virus, we experimented with two models. first model is based on ct-bert and second model is an ensemble of ct-bert, roberta and svm with tf-idf. the pre-processing steps improved the results. the performance on the validation set using pre-processing steps is reported in ## conclusion in this work, we present our models to identify covid-19 related informative tweets. we treat this as binary text classification problem. we propose two models based on pre-trained language models for this task. our model based on ct-bert achieved f1-score of 88.87%.