Linguist Geeks on WNUT -2020 Task 2: COVID -19 Informative Tweet Identification using Progressive Trained Language Models and Data Augmentation (2020)

Abstract: Since the outbreak of COVID-19, there has been a surge of digital content on social media. The content ranges from news articles, academic reports, tweets, videos, and even memes. Among such an overabundance of data, it is crucial to distinguish which information is actually informative or merely sensational, redundant or false. This work focuses on developing such a language system that can differentiate between Informative or Uninformative tweets associated with COVID-19 for WNUT-2020 Shared Task 2. For this purpose, we employ deep transfer learning models such as BERT along other techniques such as Noisy Data Augmentation and Progress Training. The approach achieves a competitive F1-score of 0.8715 on the final testing dataset.

## introduction the aim of wnut 2020 task 2 @xcite , is to produce methods that automatically classify whether an english tweet associated with the novel coronavirus or covid-19 is informative or not. an informative tweet may report information regarding recovered, suspected, confirmed and death cases or may include the knowledge of location or travel history of such occurrences. to accomplish such a system, we are provided with a dataset of 10,000 tweets, consisting of 7000 tweets for training, 1000 tweets for a validation set and 2000 tweets for the evaluation phase. our solution employed an ensemble of pretrained models such as bert @xcite , fine-tuned earlier on the task dataset, see section 2. we also investigated text augmentation techniques such as replacing tokens with synonyms, random removal and swapping of tokens, see section 3. the work also explored progressive training of a given model see section 4. certain aspects of these methodologies seemed to perform well and are discussed in detail. ## related work sequence labelling or text classification is one of the primary tasks in computational linguists. in general, the task contains different levels of scope such as document level, paragraph level, sentence level and sub-sentence level (words or groups of words). typical pipeline for sequence labelling consists of feature extraction, followed by dimensionality reduction and a classification technique. initial approaches for feature extraction involved techniques such as term frequency-inverse document frequency (tf-idf) @xcite , word2vec @xcite or global vectors for word representation (glove) @xcite . dimensionality reduction can help in reducing time and memory complexity if datasets contain a large vocabulary of unique words. therefore, this step is sometimes left out nonetheless, prevalent methods for feature extraction include principal component analysis (pca), linear discriminant analysis (lda) or t-distributed stochastic neighbor embeddings (t-sne). early models for classification such as naive bayes, and support vector machines have now been superseded by deep learning models. such models incorporate sequential processing ability through architectures such as recurrent neural networks which can be worked with a varied length of text sequences as well. lately, transfer learning has taken over these pipelines, producing high-quality textual representation by employing enormous corpora used in pre-training. deep learning methods have produced dominant performances in many ## transfer learning ensemble in our approach, we apply models like bert that have already been pre-trained on data related to covid-19. for this purpose, we used huggingface's transformers package @xcite . model acc. f1-score ct-bert 90.2 0.893 mrm8488/bioclinicalbertfinetuned-covid-papers 1 88.4 0.872 deepset/covid_bert_base foot_1 87.0 0.860 majority of the models that were tested are some variations of bert. they vary in terms of data that have been fine-tuned on or number of parameters such as bert-base or bert-large (model architecture variants for bert). for instance, deepset/covid_bert_base 2 is fine-tuned on cord-19 dataset whereas ct-bert @xcite ) is trained on crowbreaks dataset @xcite . in our system, we fine-tuned these models to the data in shared task 2 of wnut 2020. ensemble inference is a very popular trick in machine learning competitions. we employed an averaging ensemble of the models, to get an improved f1-score and accuracy on the validation set of 0.897 and 90.6 respectively. though marginal, we employed the ensemble strategy for our final submission as well. ## data augmentation another typical trick to enhance the performance of neural networks is to employ more training data. to this purpose, we applied data augmentation techniques that are loosely inspired by those used in computer vision. moreover, we wanted to add some noise to the input text, in order to produce more robust models, particular because the tweets are ordinarily noisy due to use of emoticons, social media lingos, short forms and symbols such as #, @ or urls in them. for a given sentence in the training set, performed one of the following procedures with respect to a random probability: for implementing such augmentations, we used edward ma's nlpaug package 3 , however, we note that such techniques have been studied most recently in the paper eda @xcite . other augmentation techniques that we tried but did not use were antonym replacement and sentence augmentation with models such as gpt-2 @xcite . for instance, replacing with antonyms seemed to distort the meaning of the text unless, the random selection of words somehow resulted in some form of double negation. ## progressive training of model progressive training is one of the highly recommended techniques among deep learning practitioners, especially for its practical utility. this may involve training the model first on a smaller segment of the problem such that it training on a smaller sample of data, or priorly training on a smaller number of classes as opposed to all classes. a popular variation in image classification has been popularised by jeremy howard's fastai lectures and library @xcite , that involves first training an image classification network on smaller sized images and then gradually increasing the dimensions. similar intuition was extended for training of generative adversarial networks by the progans @xcite study. inspired by the concept, we developed a similar method by performing the following steps: transformer head: consisting of a bert architecture b. classifier network: usually consists of dense or linear layers. 2. initially, the entire model is trained on smaller sized encoded vectors of the tweets, having length 128. 3. this is followed by fitting the same transformer head over encoded vectors having length 192, with an uninitialized classifier network 4. the above steps can be iterated up till the encode input length is 256. tokenized vector length acc. f1-score 128 90.2 0.893 192 91.13 0.905 256 91.63 0.913 in our work, we followed the above described method gaining a marginal improvement as shown in ## results we made 2 submissions on the test set involving the following: a. ensemble of various bert models trained over original inputs. b. progressively trained digital epidemiology lab's ct-bert model over augmented inputs. from the results on the leaderboard, submission b. scored better with an f1-score of 0.8715. ## conclusion we have outlined the motivation, design, and results of the wnut shared task 2 on detection of informative tweets pertaining to covid-19 pandemic. we employed techniques such as transfer learning, ensemble learning, data augmentation and progressive training. however, we hope to study the task and come up with new findings in the future. one of the major aspects, we would like to analyze is the timestamp of the tweets. covid-19 has had a very dynamic impact on social media, with varying trends concerning the number of cases, information regarding vaccine trials or new developments in social distancing guidelines. therefore, in further work, we would like to include a temporal factor associated with these tweets for more reliable prediction on their informativeness. furthermore, cross-lingual research and applications such as xlm-roberta (conneau et. al, 2020) have been rising in nlp studies. therefore, data augmentations from multi-lingual or code-mixed tweets and microblogs sources about covid-19 can comprehensively contribute to robust of the system such as ours. lastly, to further increase the robustness of the model, we would like to explore techniques such as knowledge distillation.