E dinburgh NLP at WNUT -2020 Task 2: Leveraging Transformers with Generalized Augmentation for Identifying Informativeness in COVID -19 Tweets (2020)

Abstract: Twitter has become an important communication channel in times of emergency. The ubiquitousness of smartphones enables people to announce an emergency they’re observing in real-time. Because of this, more agencies are interested in programatically monitoring Twitter (disaster relief organizations and news agencies) and therefore recognizing the informativeness of a tweet can help filter noise from large volumes of data. In this paper, we present our submission for WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets. Our most successful model is an ensemble of transformers including RoBERTa, XLNet, and BERTweet trained in a Semi-Supervised Learning (SSL) setting. The proposed system achieves a F1 score of 0.9011 on the test set (ranking 7th on the leaderboard), and shows significant gains in performance compared to a baseline system using fasttext embeddings.

## introduction in late december 2019, there was an identification of an outbreak of a novel coronavirus causing covid-19. foot_0 due to the rapid spread of the virus, the world health organization declared a state of emergency. among several social media platforms, twitter provides a powerful lens for identifying people's behavior, decision-making, and sources of information before, during, and after wide-scope events, such as natural disasters @xcite . due to the low signal-to-noise ratio, identifying relevant information in tweets is a challenging task. the basic goal of wnut-2020 task 2 (nguyen et al., 2020) is to automatically identify whether a covid-19 english tweet is informative or not. such informative tweets provide information about recovered, suspected, confirmed, and death cases as well as the location or travel history of the cases. about 4m covid-19 english tweets are daily being posted on twitter, most of which are, however, not informative. in many scenarios, it is not always clear whether a person's tweet is announcing a disaster response. consider an example of an uninformative tweet from the dataset as shown in figure 1 . however, this observation is near inscrutable examining only the vocabulary used; the tweet contains a variety of top frequent informative words ("coronavirus", "covid-19", "deaths"). this example hints that in order to reach meaningful results, we may have to examine contextual linguistic features, model the annotator's bias, introduce adversarial examples, and so on @xcite . in this paper, we build an ensemble of transformer @xcite models to leverage its strength in capturing contextual information. the data used to train these models is an augmented version carefully designed to alleviate the confirmation bias and thereby improve generalization. the final inference result is the majority voting of the class from all the constituent models through optimal thresholding as a post-processing step. our best model (ensemble) achieves f 1 scores of 0.9248 and 0.9011 on the development and test set respectively. ## related work recently, research has started to investigate the use of deep learning in the area of disaster response. for instance, @xcite use cnn to detect informative messages in data from floodrelated disasters and report significant improvements in performance over svm and fully connected neural networks. nguyen et al. (2017) use cnns to capture the most salient n-gram information on situational awareness crisis data and note the improvements over conventional algorithms. lazreg et al. (2016) use lstm network to learn a model from crisis tweets to generate snippets of information for summarizing the tweets. wang and lillis (2019) classify crisis-related tweets using elmo contextual word embeddings, whereas ma (2019) use a monolingual bert-based model for tweets classification problem in the disaster management field. text classification generally consists of two processes -an encoder that converts textual inputs to numerical representations and a classifier that estimates hidden relations between these representations and class labels. the text representations are generated using n-gram statistics @xcite , word embeddings @xcite . more recently, powerful pre-trained models for text representations, e.g. bert @xcite , have shown stateof-the-art performance on text classification tasks using only the simple classifier of a fully connected layer. ## system description we formulate the task of identifying informativeness in tweets as a binary text classification problem with informative and uninformative as the class names. as shown in figure 2 , the framework of our informativeness classification model consists of three modules -transformer and bertweet ensemble learning, generalized augmentation via pseudo-labeling, and optimal thresholding via post-processing to adjust the distribution of class labels in target. ## experimentation we rely on the dataset provided by the organizers to perform our experiments. overall, there are a total of 10k tweets split in the ratio of 70/10/20 parts into train/development/test set respectively. however, for the final evaluation, 12k unlabeled noisy tweets are provided, out of which 2k test tweets are the actual ones the models are finally evaluated upon. ## results and error analysis we conduct ablation analysis to compare the performance of our model variants. we evaluate the effect of contextual features by comparing our model with and without augmentation. understandably, the fine-tuned roberta model did outperform every other experimented models. bagging the models also lead to lower variance and robust predictions. the confusion matrix of our best model is as shown in figure 4 . we look through the examples where our model made misclassification, and summarize the patterns of these error examples along with their attention visualization @xcite . ## conclusion we adopt an ensemble approach to reduce the variance of predictions and improve the model performance. the empirical results show the effectiveness and robustness of our model. additionally, we perform a linguistic error analysis to gain insights into the model behavior. in the near future, we would like to combine user-related tweet features (followers, friends, favorite counts) and tweet-related meta-features (retweets, creation date, sentiment) along with the contextual representation. moreover, extending to multilingual tweets (chowdhury et al., 2020) is a potential future direction to pursue.