Phonemer at WNUT -2020 Task 2: Sequence Classification Using COVID T witter BERT and Bagging Ensemble Technique based on Plurality Voting (2020)

Abstract: This paper presents the approach that we employed to tackle the EMNLP WNUT-2020 Shared Task 2 : Identification of informative COVID-19 English Tweets. The task is to develop a system that automatically identifies whether an English Tweet related to the novel coronavirus (COVID-19) is informative or not. We solve the task in three stages. The first stage involves pre-processing the dataset by filtering only relevant information. This is followed by experimenting with multiple deep learning models like CNNs, RNNs and Transformer based models. In the last stage, we propose an ensemble of the best model trained on different subsets of the provided dataset. Our final approach achieved an F1-score of 0.9037 and we were ranked sixth overall with F1-score as the evaluation criteria.

## introduction up till mid-june 2020, the coronavirus pandemic has caused 445k deaths and has infected more than 8.2m people belonging to 215 regions and countries. this has led to a surge of panic and fear among people all around the globe. recently, there has been rapid development in building monitoring systems (e.g. the johns hopkins coronavirus dashboard) to track any news regarding the outbreak, to let users know of any information related to the coronavirus for example, new cases emerging near the user's location. most of the official sources from where the information is released are not updated very frequently and the information reported by such organizations may be stale, for example, who updates the information regarding the virus only once a day. these monitoring systems tend to use social network data like posts from facebook or tweets from twitter as an alternative source of information relating to the pandemic, generally by scraping relevant information or by crowd sourcing. however, due to increased panic and emotions among people, social media is flooded with massive amounts of data, e.g. about 4m covid-19 english tweets are posted daily on twitter. but the major issue regarding this is that the majority of such tweets are uninformative and thus, do not impart useful information that can be used. hence, any application that requires to update the news has to first filter these tweets to detect which among the tweets are actually useful. manual approaches for this filtering task are not only cumbersome, frustrating and ineffective for vast amounts of data, but also costly. this calls for automated systems which can filter the information given huge amounts of mixed data, and thus serve as the motivation for the shared task. although a lot of work has been done on the lines of sequence classification on general english texts @xcite , since the covid-19 outbreak has grown in a short while, not many systems which deal with covid-19 related texts have been developed. in this paper, we describe our approach to tackle the wnut 2020 shared task 2. the paper is structured as follows: section 2 talks about the problem statement and provided dataset. section 3 describes a step-by-step methodology process that we employ. section 4 explains the experiments that were carried out along with a detailed discussion of the dataset, system settings and results of our experiments. section 5 provides a brief conclusion of the paper along with the future scope of our research. the wnut-2020 shared task 2 (nguyen et al., 2020) is based on a sequence classification problem wherein the aim is to identify whether an english tweet related to the novel coronavirus (covid-19) is informative or not. a tweet is said to be informative if it provides information regarding recov-total positive negative training set 7000 3303 3697 validation set 1000 472 528 test set 2000 944 1056 to achieve the goals of the shared task, a dataset of 10k covid-19 english tweets are provided, in which 4719 tweets are labelled informative and 5281 tweets are labelled uninformative. each tweet is annotated by three independent annotators with an inter-annotator agreement score of fleiss' kappa at 0.818. the 10k dataset is divided into training/validation/test sets in the ratio 70/10/20 with distribution as shown in systems are evaluated using standard evaluation metrics, including accuracy, precision, recall and f1-score. however, the submissions are ranked by f1-score. ## methodology we have split the proposed methodology in three steps-data preprocessing, deep learning models for sequence classification and ensemble process (bagging). the code corresponding to each of the steps has been made available online foot_0 to facilitate further research. ## experiments we experiment with ten deep learning models with the provided training and validation splits. based on the scores produced above, we evaluate the final test predictions by training the best performing model with the seven synthesised randomly shuffled versions of the dataset, followed by merging the output predictions made on the test set. in this section, we present the dataset distribution, experimental settings, evaluation metrics, results and a brief analysis of the proposed system. ## conclusion and future work in this paper, we provide a detailed description of our approach to solve the emnlp wnut-2020 shared task 2. our approach involves processing in three stages. in the first stage, we pre-process the provided dataset by cleaning and extracting only relevant information from the provided text. in the second stage, we experiment with several deep neural networks like cnn, rnns and transformer based networks like xlnet, xlm, bert and its different variations. in the final phase, we introduce the idea of ensemble learning (bagging) to our solution which is a major improvement over the individual model. we submitted an ensemble and an individual system based on the ct-bert model as our final entries to the shared task. the ensemble approach fetches us a private leaderboard rank of 6 with f1-score as the evaluation criteria. our system promotes the usage of transformer based models pre-trained on relevant corpora and ensemble learning with as many candidate models as feasible. in future, we aim to explore the usage of non-ascii characters like emoticons as features for classification and imparting ensemble learning through an end-to-end deep learning solution.