DATAMAFIA at WNUT -2020 T ask 2: A S tudy of P re-trained L anguage M odels along with R egularization T echniques for D ownstream T asks (2020)

Abstract: This document describes the system description developed by team datamafia at WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets. This paper contains a thorough study of pre-trained language models on downstream binary classification task over noisy user generated Twitter data. The solution submitted to final test leaderboard is a fine tuned RoBERTa model which achieves F1 score of 90.8% and 89.4% on the dev and test data respectively. In the later part, we explore several techniques for injecting regularization explicitly into language models to generalize predictions over noisy data. Our experiments show that adding regularizations to RoBERTa pre-trained model can be very robust to data and annotation noises and can improve overall performance by more than 1.2%.

## introduction the recent outbreak of coronavirus disease has turned the world topsy-turvy with more than 25m+ infected people so far and 800k+ deaths across the globe 1 . government officials, researchers, health workers and fear trapped common people are largely relying on online information to monitor, tackle and overcome the situation. social media platforms, particularly -twitter and facebook, have become an easily accessible source of information related to the current affairs. very recently, few researchers (drias and drias, 2020; @xcite have conducted large scale analysis on twitter data in the context of covid-19. however, as mentioned by @xcite , a huge majority of the information shared on twitter are not informative and can pose an additional burden to those who are relying on social media to monitor the pandemic. for example -a tweet like "half of uruguay's covid-19 cases can be traced 1 https://covid19.who.int/ to a single fashion designer" can be speculative and possibly does not contain any insightful information. on the other hand, a tweet like "currently 32000+ deaths and their talking spreading it far and wide...bbc news -coronavirus: trump unveils plan to reopen states in phases" can be very useful to a larger population. to overcome this situation, shared task 2 of wnut 2020 by @xcite allows to automatically identify whether a tweet is informative in the context of covid-19 or not. the task dataset contains 10k tweets (written mostly in english) and the associated label -informa-tive and, uninformative labelled by human annotators. in this task, we use a fine-tuned pre-trained roberta base model @xcite to learn the contextual representation of texts. we discover further that the last 4 layers of roberta contain semantically rich hidden representation and are diverse, which, when used together can lead to better performance. in our final submitted model, as described in section 2.1, we use the concatenated hidden states of all the tokens from the last 4 layers of roberta base . upon further investigation, we realize that the overparameterized large transformer models can be prone to overfitting when fine-tuned on noisy data and ambiguous annotations. in the later part of our study (section 2.3), we explore various different techniques to inject regularization externally to pre-trained language models to improve generalization capabilities. although, ensembling diverse set of classifiers @xcite is known to be an effective technique for improving generalization, in real-life applications, large ensemble systems aren't much effective for drawing inferences on low-resource devices. further, interpreting model predictions are also difficult for complex ensemble systems. due to these operational challenges, we refrain ourselves from using ensem-bles and rather focus on single-model systems in our work. we have open-sourced our system and the experiments at github foot_0 . ## system description in this shared task we use the original train, dev and test datasets provided in the challenge foot_1 . dev dataset is used for only validation and evaluating our models. we omit the description of the datasets in this paper due to page constraint, and it can be found in the task description by nguyen et al. 2020b. ## results we evaluate the performances of all the models using f1, precision and recall scores. ## result analysis in this section, we inspect the language models and explain their predictive capabilities. in exploratory data analysis (eda), we plot top words present in the corpus, conditioned on the informative and uninformative classes and figure out that "case", "covid", "death", "virus" etc. remains top words for both the classes of documents. we observe that any language model in just 2-3 epochs of fine-tuning can achieve a f1 score of more than 89%, however, due to the inherent noise of the tweets, around 10% of the examples are ambiguous and difficult to be classified correctly for almost all the standalone models. figure 3 shows the two different clusters of text representations extracted by system model (embedded onto a lower dimensional space) on the dev dataset, with several misclassified ambiguous examples. from ## conclusion in this paper, we present a large-scale empirical study of language models with explicit regularizations. we conclude that using hidden states from multiple layers from a language model helps in understanding the context better and further using an additional regularization, we can improve the stability and generalization capabilities of large pre-trained models. in future, we wish to use the insights captured by this work in building a custom and robust language model particularly for noisy user generated texts that are found in social media. another interesting extension would be to prove the theoretical justifications and calculating the generalization bounds for each of the explored regularization methods. we strongly believe that our study will help the research community in using language models on real-life applications more effectively.