NLP North at WNUT -2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID -19 E nglish Tweets (2020)

Abstract: With the COVID-19 pandemic raging world-wide since the beginning of the 2020 decade, the need for monitoring systems to track relevant information on social media is vitally important. This paper describes our submission to the WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets. We investigate the effectiveness for a variety of classification models, and found that domain-specific pre-trained BERT models lead to the best performance. On top of this, we attempt a variety of ensembling strategies, but these attempts did not lead to further improvements. Our final best model, the standalone CT-BERT model, proved to be highly competitive, leading to a shared first place in the shared task. Our results emphasize the importance of domain and task-related pre-training.

## introduction the amount of covid-19 pandemic cases is rapidly approaching 25m world wide, with almost 1m people who have lost their life to the merciless disease, according to worldometer. 2 this paper exploits the capabilities of natural language processing (nlp) techniques to extract informative tweets, and is a participation in the wnut-2020 task 2: identification of informative covid-19 english tweets @xcite . social media is a useful medium for rapid access to information about the pandemic -but along with all the informative tweets comes an even larger amount of non-informative information. being able to extract what is informative, and hereby leave out all the non-informative posts, is vital in monitoring and tracking the development of covid-19. in the informative oklahoma's first confirmed case of coronavirus is in tulsa @xmath0, aids, and cancer in the same day and the media would say he wasn't doing anything. shared task, informative tweets were defined as to contain information about covid-19 cases such as statistics, locations or travel history. figure 1 shows two examples from the training data. the introduction of neural networks has led to an increase in performance for many natural language processing tasks @xcite . however, previous work on classification showed that svms with character and/or word n-grams often still outperform neural networks @xcite @xcite . neural network approaches can elegantly exploit raw data, by pre-training word embeddings using a language modeling objective. recently, more powerful contextual embeddings were introduced @xcite , which base each word embedding on its context. these contextual embeddings are generally pre-trained on huge amounts of raw data, and then fine-tuned on the target task. this leads to the question: how do the three types of classification models viz. svm, neural models with pre-trained embeddings and various contextual models compare and perform in this classification task? (rq1) neural networks as well as transformer-based models can directly exploit additional raw data by pre-training. this pre-training often leads to superior performance, depending mainly on the size and distribution of the pre-training data. although no additional annotation effort is necessary for pretraining, it often comes with huge computational cost and exhaustive training time. recent work has shown that selecting data which matches the target domain better (domain-specific or task-specific) is important for transformer-based pre-training @xcite . this leads to the question: how important is task-specific pretraining for detection of informative covid-19 tweets? (rq2) finally, we are interested in the supplementary of the three different architectures. even though one model outperforms the other two models, it can still be that they have different strengths, and combining them can thus lead to superior performance. our last question is: can we ensemble svm, neural network and bert-based models to improve robustness? (rq3) 2 methodology below we will discuss our implementations of each of the classifiers and the ensemble models. ## conclusion in this paper we have presented our winning participation for the shared task of wnut-2020 on identification of informative covid-19 english tweets. we evaluated three types of models; svm, neural networks with pre-trained embeddings, and transformer models. we found that the transformerbased covid-related ct-bert model performed the best, achieving an f1 score of 90.96 on the hidden test data (rq1). evaluating our models on the development data, we found that the ct-bert model, which was pre-trained on domain and taskrelated data, performed better than bert-base and roberta pre-trained on data unrelated to the shared task (rq2). different types of ensembling approaches were tested in an attempt to improve robustness. this included majority voting, stacking and thresholding. we found stacking to be most competitive, albeit it still underperformed compared to standalone ct-bert (-0.55%) (rq3).