UET at WNUT -2020 Task 2: A Study of Combining Transfer Learning Methods for Text Classification with R o BERT a (2020)

Abstract: This paper reports our approach and the results of our experiments for W-NUT task 2: Identification of Informative COVID-19 English Tweets. In this paper, we test out the effectiveness of transfer learning method with state of the art language models as RoBERTa on this text classification task. Moreover, we examine the benefit of applying additional fine-tuning and training techniques including fine-tuning discrimination, gradual unfreezing as well as our custom head for the classifier. Our best model results in a high F1-score of 89.89 on the task’s private test dataset and that of 90.96 on public test set without ensembling multiple models and additional data.

## introduction identification of informative covid-19 english tweets @xcite is the task of "providing users the information related to the virus". it is meaningful in the sense that with an increasing amount of tweets about the virus, many among them are uninformative and even harmful to the viewers. manually identifying uninformative tweets is costly. therefore, a system which can perform the task automatically would be tremendously helpful. with the rise of deep learning, particularly transfer learning for solving text classification problem, we would like to propose an approach that leads to high performance (represents by a high f1-score of 89.89) on the task's test dataset. this approach uses pre-training method with state-ofthe-art pre-trained language model roberta @xcite , combines with many existing finetuning techniques including one-cycle-policy learning rate @xcite ,fine-tuning discrimination, gradual unfreezing @xcite , label smoothing @xcite , and our custom-head model. other than using pre-training with a state-of-theart pre-trained language model, there is no clear winning factor for our success as all fine-tuning techniques need to incorporate to form our best model. our main contributions are: ## conclusion in this paper, we experiment with the effectiveness of transfer learning using state-of-the-art language pre-trained model roberta with the incorporation of several fine-tuning and training techniques for the informative tweet identifying the task. our best model results in high f1-scores in both public and test dataset. for future work, we would like to explore the effectiveness of different text augmentation strategies and task-adaptive pre-training instead of only fine-tuning the classification task.