L ynyrd S kynyrd at WNUT -2020 Task 2: Semi-Supervised Learning for Identification of Informative COVID -19 E nglish Tweets (2020)

Abstract: In this work, we describe our system for WNUT-2020 shared task on the identification of informative COVID-19 English tweets. Our system is an ensemble of various machine learning methods, leveraging both traditional feature-based classifiers as well as recent advances in pre-trained language models that help in capturing the syntactic, semantic, and contextual features from the tweets. We further employ pseudo-labelling to incorporate the unlabelled Twitter data released on the pandemic. Our best performing model achieves an F1-score of 0.9179 on the provided validation set and 0.8805 on the blind test-set.

## introduction as of october 8, 2020, the novel coronavirus, sars cov-2, has infected 35.8 million people and led to over 1 million deaths 1 . with several stay-athome orders and lockdowns in place, people have spent considerably more time than usual in disseminating and consuming information through social media. a subset of this publicly shared information relates to covid-19, and ranges from information about new cases and deaths, lapses in policies made by local and state agencies, denied access to testing, etc. since well-designed tools that monitor such information can help in quick identification and response from concerned agencies, recent research has focused on automatically detecting information and events related to covid-19. for instance, @xcite aim to identify events on twitter that relate to a new reported case or death, lapses in testing facilities, and information about cure and prevention, @xcite analyze tweets to understand the perceptions of covid-19 policies and how they change over time. in the same vein as above, the 2020 edition of workshop on noisy user-generated text (w-nut 2020) hosted a shared task on 'identification of informative covid-19 english tweets'. the task involves automatically identifying whether an english tweet related to covid-19 is 'informative' or not. for a tweet to be considered informative in this context, it should provide information about recovered, suspected, confirmed, and death cases as well as location or travel history of the cases. the goal for developing such an automated system is to help track the development of the covid-19 outbreak and to provide users the information related to the virus, e.g. any new suspicious/confirmed cases near/in the users' regions. aligned with the goals of this shared task, our paper details the use of state-of-the-art natural language processing techniques for identifying informative covid-19 tweets. we experiment with a variety of methods, ranging from feature-based classifiers to leveraging recent advances in pretrained neural architectures (section 3). to further improve performance, we incorporate unlabelled tweets released on covid-19 via masked language modelling and pseudo-labelling techniques. our best performing model is an ensemble that uses logistic regression to combine the output probabilities of several base classifiers (section 4). we further analyze the impact of pre-processing and semisupervision through ablation studies. through our qualitative adversarial analysis, we show how the predictions of bert model are sensitive towards specific tokens such as 'confirmed case' or even locations and numerals, which also guides our data pre-processing steps. ## data preprocessing we conduct classification experiments on the covid-19 tweets dataset @xcite provided for the shared task. the data split consists of 7000 tweets for training, 1000 in the validation set and 2000 in the blind test-set. we start by lowercasing all the tweets, and replacing all urls with 'httpurl' and all usernames with '@user'. we then normalize all characters and pictograms. additionally, we remove bad symbols (e.g. from a different language), any html tags, duplicated symbols or characters (like dots, question marks, special symbols, dashes or exclamations), and unnecessary underscores in the tokens. we also isolate punctuations, expand contractions (e.g. there'll to there will), and replace leet alphabets with their correct english versions using leet vocabulary foot_0 . we convert emojis into their text form foot_1 and finally normalize the different ways in which 'covid-19' is written (e.g., covid-19, covid2019, covid19, or covid 2019) by replacing each occurrence with 'covid19'. these steps help to limit the vocabulary for better learning. on top of the above steps, we create different versions of the dataset as described below. cleaned: no further processing is done. num-replaced: all the numerals in the dataset (except 19 in 'covid19') are replaced by num followed by the number of digits in the number. loc-replaced: we use the python spacy library foot_2 to get the named entity tags for each token in a tweet and replace the token having a 'gpe' (geopolitical entities i.e. countries, states, or cities) tag with 'loc', so that the informativeness of a tweet is agnostic of a particular location. num-loc-replaced: we replace both numerals and tokens with 'gpe' tags in this version. the motivation behind different versions of the dataset comes from our analysis (presented in section 4) of the model trained on the cleaned version of the dataset. we show how the model predictions are sensitive towards locations and numerals in a tweet, as a result of the patterns in the dataset. hence, using num and loc helps mitigate these issues. following this pre-processing, we train a variety of machine learning models to classify tweets into two classes: informative and uninformative. we use the data split provided by the task organizers throughout to ensure consistency. ## system description the system submitted for the shared-task is an ensemble of a variety of machine learning models which we discuss below in detail. ## results and analysis we first illustrate the benefit of employing pseudo-labelling in figure 1 . all three methods, namely hcf, use and ngrams achieve better f1scores by leveraging the unlabelled data, along with the provided training dataset. hence, we next compare these models with other baselines and bertbased pre-trained methods in input: the taslee palm city estate in maitama , abuja , has alerted .. ## conclusion in this paper, we describe our system to identify informative covid-19 english tweets. we find that an ensemble model which uses a logistic regression to combine the predictions of a variety of feature-based to neural methods achieves the best performance on the shared task. our analysis shows that incorporating unlabelled tweets results in consistent performance gains. we show how the trained model can be sensitive to specific tokens in the tweets, and hence, advice for exercising caution while deploying machine learning models for downstream monitoring applications.