Not- NUT s at WNUT -2020 Task 2: A BERT -based System in Identifying Informative COVID -19 E nglish Tweets (2020)

Abstract: As of 2020 when the COVID-19 pandemic is full-blown on a global scale, people’s need to have access to legitimate information regarding COVID-19 is more urgent than ever, especially via online media where the abundance of irrelevant information overshadows the more informative ones. In response to such, we proposed a model that, given an English tweet, automatically identifies whether that tweet bears informative content regarding COVID-19 or not. By ensembling different BERTweet model configurations, we have achieved competitive results that are only shy of those by top performing teams by roughly 1% in terms of F1 score on the informative class. In the post-competition period, we have also experimented with various other approaches that potentially boost generalization to a new dataset.

## introduction following the rise of smart technology and an increasingly wide coverage of internet, social network websites are becoming ubiquitous these days. besides serving as a platform for various types of entertainment, social media is particularly helpful in spreading information, and such can be leveraged to keep the majority of its users wellinformed amidst a natural disaster or a pandemic like covid-19. one major advantage of sourcing information via social media is that all information is updated in real-time. any person with a social media account can post or share information instantly at the moment he/she witness a noteworthy event. this is a much faster way to obtain information compared to reading newspaper, watching the news on tv, or viewing other official source * equal contribution with the first author of information since most tend to be updated only at mid-day or at the end of day. nevertheless, information on social media platforms is mostly not verified, heavily opinionated towards the person who posted it, and at worst, completely inaccurate. this highlights the need for a system that can automatically identify legitimate information from the huge pool of information. in order to address the aforementioned need for such a system, in this paper we attempt to tackle the wnut 2020 task 2: identification of informative covid-19 english tweets @xcite . as stated in the task's description paper, this task requires its participants to build and refine systems that, given an english tweet carrying covid-19related content, automatically classify whether it is informative or not. in the context of this shared task, being informative is defined as bearing information regarding suspected, confirmed, recovered or death cases related to covid-19 as well as location or travel history of these cases. ## related work text classification is a simple but practical task in the field of natural language processing. early models such as naive bayes, logistic regression, and support vector machine are widely known and used as a headstart for experimenting classification tasks due to their simplicity and fast training time while still able to achieve a reasonable performance. the rise of modern neural network brings deep learning to the classification tasks within the language processing field as it helps induce features for learning. further development of recurrent networks gives us the ability to deal with sequences of varied lengths, which improves the performance of text classification to a great extent. while classifying texts, it is essential to make the machine understand deeply the characteristics of input sequences. because of that, having a well-performing system that embed text sequences is an important prerequisite in building a good model for text classification. recently, pre-trained language models let us achieve high quality text embeddings, which then can be used for further downstream tasks. for language processing, the most famous pre-trained contextual language models recently are bert @xcite , elmos @xcite , and xl-net @xcite . ## system description we use the pre-trained language model bertweet (nguyen et al., 2020a), an english tweet domainspecific model inspired by the original bert model @xcite , as the core for our system (more details will be discussed later). to accomplish the task of identifying informativeness of covid-19 english tweets, we attach a classification block on top of our bertweet block, which is a combination of one or more linear layers. figure 1 indicates the high level detail of our system. 3.1 bertweet bertweet (nguyen et al., 2020a) is a large-scale language model pre-trained for english tweets. because of its nature of being a domain-specific model, bertweet has achieved state-of-the-art performances on many downstream tweet nlp tasks such as part-of-speech tagging, named entity recognition, and text classification, outperformed top models such as roberta-base (liu et al., 2019) and xlm-r-base (conneau et al., 2019). trained on 845m tweets streamed from 01/2012 to 08/2019 and 5m tweets related the covid-19 pandemic as pre-training resources, bertweet has ## conclusion in this paper, we proposed a system that carries out the automatic identification of informative versus uninformative tweets. while this system is simple, it has leveraged recent advances and state-of-theart results in natural language processing and deep learning, namely bert-based models. for our future work, we will augment this system so that it can work for various forms of information circulating on social media such as facebook status, reddit post, instagram caption, etc.