UIT - HSE at WNUT -2020 Task 2: Exploiting CT - BERT for Identifying COVID -19 Information on the T witter Social Network (2020)

Abstract: Recently, COVID-19 has affected a variety of real-life aspects of the world and led to dreadful consequences. More and more tweets about COVID-19 has been shared publicly on Twitter. However, the plurality of those Tweets are uninformative, which is challenging to build automatic systems to detect the informative ones for useful AI applications. In this paper, we present our results at the W-NUT 2020 Shared Task 2: Identification of Informative COVID-19 English Tweets. In particular, we propose our simple but effective approach using the transformer-based models based on COVID-Twitter-BERT (CT-BERT) with different fine-tuning techniques. As a result, we achieve the F1-Score of 90.94% with the third place on the leaderboard of this task which attracted 56 submitted teams in total.

## introduction in the mid of april 2020, the covid-19 pandemic has caused 23m affected and more than 800,000 deaths almost over the world foot_0 , and the number of people affected and death increase day by day. the world faces many challenges from many sides. one of the biggest problems is how to prevent the spread of covid-19 before the vaccine is ready. additionally, how to provide people information about covid-19 with the fastest and most timely is one of the most issues. traditionally, the information will provide some official sources as who or news on tv, but with the incredible speed of virus spreading, this cannot be real-time update. the social network is the channel that can solve this problem, and twitter is one of the popular social networks with a massive amount of data. the 0 * these authors contributed equally to this work. number of tweets related to covid-19 on twitter is up to 4 million daily @xcite , and most of these tweets are uninformative. because of that reason, detecting which tweet is informative or not is necessary. thus, the w-nut 2020 shared task 2: identification of informative covid-19 english tweets @xcite attracted about 120 different teams coming from 20 countries. we present our solution for this task which is also our main contribution in this paper. in recent years, transfer learning models such as bert @xcite , albert @xcite , bertweet @xcite , xlm-r @xcite , xlnet @xcite , covid-twitter-bert @xcite have achieved the best performances on a variety of nlp tasks, e.g., text classification @xcite , machine reading comprehension ( @xcite . fine-tuning pre-trained language models @xcite also have proved the effectiveness of exploring solutions for new tasks. in addition, ensemble approaches were also very effective in previous studies @xcite @xcite . inspired by these research works, we propose our simple but effective ensemble strategy using different models based on ct-bert with different fine-tunings, achieving the f1-score of 90.94% on the final test set. this result reaches the third rank on the leaderboard of this competition. the rest of this paper is structured as follows. section 2 is task definition. section 3 introduces information about the dataset .section 4 describes our approach. section 5 shows experiments and results on the dataset. finally, section 6 concludes the paper and discusses future work. dataset for fine tuninig pre-processing generating various models with finetuning ct-bert selecting 7 best models ensemble model prediction 2 task definition in this section, we summarize the w-nut-2020 shared task 2 @xcite . the objective of this task is to detect whether a covid-19 text on twitter is informative or not. such informative tweet provides information about recovered, suspected, confirmed, and death cases as well as the location and history of each case. formally, the task is described as follows. ## dataset we use the dataset about covid-19 english tweets @xcite for our experiment. this corpus consists of nearly 10,000 covid tweets, including 4,719 tweets annotated as informative and 5,281 tweets annotated as uninformative. each tweet on this dataset is annotated by three independent annotators with an inter-annotator agreement score of fleiss' kappa at 81.80%. this dataset is divided into training/validation/test sets with a 70/10/20 rate. ## our approach in this section, we present our simple yet effective approach for this task. instead of leveraging multiple supper models, we only focus on the ct-bert model for generating a variety of best-performance models by fine-turning techniques. figure 1 shows the overview of the approach using three important components which are pre-processing techniques (see subsection 4.1), the core method ct-bert (see subsection 4.2), and the ensemble learning (see subsection 4.3). ## experiments in this section, we introduce our simple but effective ensemble model on the dataset. our systems are evaluated using accuracy, precision (p), recall (r) and f1-score @xcite . note that p, r and f1-score are calculated on the informa-tive class. ## conclusion and future work in this paper, we propose the simple but effective ensemble approach to achieve better performance on a new dataset about covid-19 based on tweets on twitter and perform experiments to compare ct-bert performances with different random seed and epoch. the experimental results showed that our proposed ensemble method with combination of data clean is effective for this dataset. our best performance reaches 93.19% in f1 on the validation set and 90.94% on the test set. although our model performance achieves rank the third place, there is just a little gap between our model and the best model (a really small difference of 0.02%). in the future, we plan to combine more strong transfer learning models @xcite with different parameters to obtain higher performances. as our experiment, our ensemble model still does not reach the highest performance, so if we try with other different parameters, it will make different results. in addition, because we use max sequence length of ct-bert is 96 tokens to fit with pre-trained data of ct-bert, a small number of sentences that are not short enough will be truncated. we will try to analyze and handle these cases.