D artmouth CS at WNUT -2020 Task 2: Informative COVID -19 Tweet Classification Using BERT (2020)

Abstract: We describe the systems developed for the WNUT-2020 shared task 2, identification of informative COVID-19 English Tweets. BERT is a highly performant model for Natural Language Processing tasks. We increased BERT’s performance in this classification task by fine-tuning BERT and concatenating its embeddings with Tweet-specific features and training a Support Vector Machine (SVM) for classification (henceforth called BERT+). We compared its performance to a suite of machine learning models. We used a Twitter specific data cleaning pipeline and word-level TF-IDF to extract features for the non-BERT models. BERT+ was the top performing model with an F1-score of 0.8713.

## introduction in an effort to aid automated the development of covid-19 related monitoring systems, the wnut-2020 shared task 2: identification of informative covid-19 english tweets tasked participants with developing systems to automatically classify tweets as informative or uninformative. the wnut task organizers have constructed and provided a data set of 10,000 tweets related to covid-19 for this task. @xcite . for this language processing task, we used google's bidirectional encoder representations from transformers (bert) to achieve performant results. bert uses the now ubiquitous transformer neural network architecture as explained in depth in the article, "attention is all you need," @xcite and garnered acclaim for obtaining new state-of-the-art results on eleven natural language processing tasks, including pushing the glue score to 80.5%. @xcite to optimize bert for this task, we finetuned the bert-large-uncased pretrained language model on the wnut-2020 shared task 2 data set. we then further improved performance by concatenating the fine-tuned bert embedding vectors with tweet-specific features and using a support vector machine (svm) for classification (bert+). to benchmark the performance of the bert+ model, we compared its performance to five traditional classifiers. we also developed a preprocessing pipeline for data cleaning and used text frequency inverse document frequency (tf-idf) to extract features for the traditional classifiers. ## preprocessing to clean the raw tweets we created a data processing pipeline to: 1) remove non-alphanumeric characters, 2) remove stop-words, 3) convert words to their lemmas, and 4) convert words to lower case. we used the natural language toolkit (nltk) python package @xcite ) for these methods. to handle the unique lexicon of twitter, we implemented additional preprocessing methods to: 1) remove @user tokens, 2) remove httpurl tokens, 3) remove the "#" character, 4) compress repeated characters, and 5) represent emojis as words. the data set provided replaced urls and in-tweet mentions of other users with the httpurl and @user tokens respectively. our data-cleaning pipeline removes these tokens when cleaning the data to avoid the models over-fitting to these tokens that do not reflect their original usage within the tweet. similarly, we removed the # character from tweets. we converted emojis into word tokens so that the models would interpret the emojis as words. when using bert, this allowed the bert model to generate word embeddings for these emojis tweets occasionally contain repeated characters with the purpose of emphasizing a word. for example, "yesssss" instead of "yes". in order to consistently capture this kind of emphasis as a separate feature from the original word, we compressed the repetition of a single character into two repetitions of that character. ## results the results reported in subsections 5.1 and 5.2 were generated using 8-fold cross validation on the combined train and validation data sets. ## conclusion & future work in this paper, we have described multiple techniques for automatically identifying and classifying informative covid-19 tweets. we have demonstrated the applicability of logistic regression with an optimized data cleaning pipeline and tf-idf for feature extraction for the task of tweet classification. we have also displayed the higher performance of the bert+ model. automated classification of real time data feeds will be important as the covid-19 pandemic continues to impact the world around us. for future work, we would to like pre-train a bert model on a large corpus of tweets as twitter's lexicon and grammatical styling differ from normal usage of the english language. we also want to compare the performance of our fine-tuned bert model to the performances of other stateof-the-art, pretrained nlp models such as fast.ai's ulmfit @xcite ) and openai's gpt2 @xcite on this task. we would also like to train a convolutional neural network for tweet classification. moreover, as @xcite has demonstrated, convolutional neural network (cnn) trained on top of pre-trained word vectors can achieve state-of-the-art performance for sequence classification. we would like to develop a similar method of utilizing a cnn for the task of tweet classification.