N ut C racker at WNUT -2020 Task 2: Robustly Identifying Informative COVID -19 Tweets using Ensembling and Adversarial Training (2020)

Abstract: We experiment with COVID-Twitter-BERT and RoBERTa models to identify informative COVID-19 tweets. We further experiment with adversarial training to make our models robust. The ensemble of COVID-Twitter-BERT and RoBERTa obtains a F1-score of 0.9096 (on the positive class) on the test data of WNUT-2020 Task 2 and ranks 1st on the leaderboard. The ensemble of the models trained using adversarial training also produces similar result.

## introduction since 2006, twitter has been a popular social network where people express their thoughts and opinions on various topics. the enforcement of lockdown in various parts of the world, due to covid-19, led to an increased usage of social media. thus, the world has witnessed a plethora of tweets since the beginning of covid-19. people have tweeted about many issues regarding the pandemic, mostly about the increasing infection rate, carelessness and incapability of governance and authorities to handle the increasing rate of cases. in addition, various preventive measures have also been conveyed through tweets. although there have been more than 600 million english tweets on covid-19 @xcite , only a few of them are informative enough to be used by various monitoring systems to update their databases. manual identification of these informative tweets can be tedious and erroneous. hence, there is a dire need to develop systems in the form of machine learning models that can help us in filtering informative tweets. in this paper, we present our approaches for the shared task "identification of informative covid-19 english tweets" organized under the workshop on noisy user-generated text (w-nut). our method makes use of ensembles consisting of bidirectional encoder representations from transformers (bert) @xcite pretrained on covid-19 tweets @xcite and robustly optimized bert pretraining approach (roberta) @xcite . we also experiment with adversarial training so as to create models that generalise well and are robust. the rest of the paper is organized as follows: related work has been discussed in section 2, followed by a brief description of the data used in section 3. the proposed methods and experimental settings foot_0 have been elaborated in section 4, 5 and 6. section 7 and 8 contains the results and error analysis respectively. section 9 concludes the paper and also includes possible future work. ## related work there has been much research to identify informative tweets during times of emergency and disaster. @xcite explored the performance of traditional machine learning algorithms and deep learning approaches for identifying informative tweets during disaster. they created manual features using the content of tweets for machine learning approaches and also tried out features obtained from convolutional and recurrent networks for deep learning approaches. a multi-modal approach for classifying informative tweets during disaster was proposed by @xcite . they combined the features obtained from text by a convolutional neural network and the vgg-16 features obtained from the image accompanying the tweet, using late fusion for better performance than models using only text or only image. @xcite proposed a classification and summarisation approach to identify informative tweets during the fani cyclone (which occurred in 2019, affecting large parts of south-east asia). they trained a support vector machine (svm) on linguistic features and parts of speech (pos) tags from tweets to identify informative tweets. to summarise the informative tweets, they experimented with latent semantic analysis (lsa) and luhn summarisation techniques. @xcite experimented with the transformer based architecture bert to classify disaster related tweets into multi-label information types. they preprocessed the tweets of trec-is dataset before feeding them into bert to produce significantly better results than the median score. in addition, they also experimented with focal loss instead of binary crossentropy loss. a new dataset for identifying informative tweet was released by @xcite . the results of various machine learning algorithms using glove @xcite word embeddings, syntactic information in the form of tf-idf vectors and bert embeddings were also presented in the work. due to the abundance of tweets related to covid-19, many works have been done to analyze the content, intent and effect of such tweets. @xcite presented an analysis of covid-19 tweets on the grounds of location, content and misinformation spread. a comprehensive study about the content of misinformative covid-19 tweets and other aspects associated with them have been done by @xcite . ## dataset the dataset @xcite provided to the participants of the shared task contains 10,000 english covid-19 tweets, out of which 4719 are labeled as informative and 5281 are labeled as uninformative. the tweets were annotated by 3 independent annotators and an inter-annotator agreement score of fleiss' kappa at 0.818 was obtained. the dataset contains the tweet id, the tweet and the corresponding label. ## data preprocessing twitter data contains a lot of noise. therefore, preprocessing on twitter data will help the pretrained models in better performance. we perform the following data preprocessing steps, most of which have been inspired from müller et al. ( 2020) : 1. unescape html tags 2. remove unnecessary spaces, tabs and newlines 3. replacing the the mentioned hyperlinks in the tweets (depicted as httpurl), with url. a simple explanation for this could be that "url" is a more commonly used expression of hyperlinks than httpurl. the user handles were already replaced by @user in the tweets, hence no processing was required. ## models we experiment with the following models and techniques: roberta is pretrained on a larger dataset as compared to bert, uses a larger batch size and replaces the next sentence prediction objective. it also uses dynamic masking pattern as a better alternative to the static masking pattern used in bert, i.e. roberta duplicates the data and masks those differently each time, whereas bert will mask the data only once. although many approaches for adversarial training in nlp have been developed, we experiment with the approach proposed by @xcite with a slight modification. in their approach, first the word embeddings are normalized. the gradients are then computed using the data and the required perturbations are created using the obtained gradients. let the sequence of (normalized) word embedding vectors of a text be t. the model parameters are represented by θ. the probability of the text belonging to class y is given by p(y|t; θ). the adversarial perturbations z adv are computed as follows: by using the gradients calculated from the above loss, the weights of the model are updated (the non-perturbed word embeddings of the model are updated). the slight modification in our experiments is that we do not normalize our pretrained word embedding of the model, since it might change the semantic meaning of the pretrained word embeddings. we perform adversarial training on both covid-twitter-bert and roberta large models @xmath0. ## experimental settings we concatenate the training and the validation data and perform a 5-fold stratified cross validation to train our models. each fold is trained for 5 epochs using early stopping with patience of 3 and tolerance of 1e-3. the models are optimised using adamw @xcite with a learning rate of 2e-5 and a batch size of 16. the models have been implemented using pytorch @xcite ) and huggingface's transformers @xcite library. ## results we evaluate the performance of the models and their ensemble using cross-validation (cv). we also tabulate the models' performance on the test set as evaluated by the f1 score on the positive class. the ensembling done at two levels increases the robustness of the results. the out-of-folds predictions were used to find the optimal threshold of the submissions, 0.498 for the ensemble without adversarial training and 0.487 for the ensemble with adversarial training. we also compare our results with the fasttext-baseline @xcite ) by the organizers. the ensemble of covid-twitter-bert and roberta large performs the best on the test data and also obtains the 1 st rank on the leaderboard. owing to its pretraining, covid-twitter-bert performs better than roberta large. the adversarial training of the models is also found to boost the scores. the ensemble of adversarial models produces results similar to the ensemble of models trained without adversarial training. ## analysis we perform our analysis on the out-of-folds predictions from our two ensembles -without adversarial training and with adversarial training. we calculate the binary cross entropy loss for all samples and then examine some of the top common mis-classifications by both the ensembles ( we observe that there are instances where our models are mistaken. a possible reason might be the variance in gold-labels of samples from annotator-to-annotator. this acts as noise in the labels of the data which is used to train the normal ensemble models. on the other hand, adversarial training adds some noise to the samples. hence, the models in the adversarial ensemble have been trained on data which has a combination of existing noise and externally added noise. we believe that because of this difference in noise, the models in the two ensembles must be behaving in different ways. to inspect this, we examine the number of samples which have been inferred incorrectly by one ensemble and correctly by the other. we observe that the normal ensemble misclassified a total of 277 samples whereas, only 260 samples were misclassified by the adversarial ensemble. moreover, out of the 277 examples that were misclassified by the normal ensemble, 102 were correctly predicted by the adversarial ensemble. on the other hand, only 85 out of the 260 samples misclassified by the adversarial ensemble, were correctly predicted by the normal ensemble. thus, it is evident that the models in both the ensembles have learnt different patterns from the data. ## conclusion we explored the performance of covid-twitter-bert and roberta-large at identifying covid-19 english tweets that are informative in nature. their ensemble achieves the state-of-the-art performance. adversarial training is found to improve our model further. for future work, we can pretrain other transformer-based models on covid-19 tweets. data augmentation techniques can help us generate more data for training models. we can also experiment with combining models trained with and without adversarial training. tweet label election judge hospitalized after primary dies of coronavirus #rip #sem-perfi fought for his life while @user got her hair done #letthatsinkin "face of chicago" thinks she's more important than chicagoans welfare #light-footliedpeopledied httpurl uninformative austin area nursing home residents who test positive for covid-19 but do not need to be in the hospital will soon be moving to one of two new "isolation facilities," one in travis county, one in williamson county: httpurl @user informative local news shoutout: have a family member or close friend with a michigan connection who has died from covid-19 and would like to share their story with @user please contact georgea kovanis at gkovanis@user informative breaking: southern ab #coronavirus case rumored to be steve busey, the younger brother of movie star @user according to our sources, steve works in the oil &amp; gas industry and is a huge @user fan. #covid19 httpurl uninformative