NEU at WNUT -2020 Task 2: Data Augmentation To Tell BERT That Death Is Not Necessarily Informative (2020)

Abstract: Millions of people around the world are sharing COVID-19 related information on social media platforms. Since not all the information shared on the social media is useful, a machine learning system to identify informative posts can help users in finding relevant information. In this paper, we present a BERT classifier system for W-NUT2020 Shared Task 2: Identification of Informative COVID-19 English Tweets. Further, we show that BERT exploits some easy signals to identify informative tweets, and adding simple patterns to uninformative tweets drastically degrades BERT performance. In particular, simply adding “10 deaths” to tweets in dev set, reduces BERT F1- score from 92.63 to 7.28. We also propose a simple data augmentation technique that helps in improving the robustness and generalization ability of the BERT classifier.

## introduction covid-19 pandemic as well as covid-19 related information both are spreading across the world rapidly. easy access to the internet made the consumption and sharing of information much faster. millions of people are sharing covid-19 related information using social media channels such as facebook, twitter. social media is divided over several issues related to masks, social distancing, covid-19 testing, etc. unfortunately, the response to the coronavirus has been often determined by people's ideology instead of health officials' guidelines. one challenge with massive information available on social media is to separate useful covid-19 related information from the noise. as a testimony, the director of who in munich security conference said "we're not just fighting an epidemic; we're fighting an infodemic" foot_0 . nlp community have taken multiple initiatives to fight this infodemic including misinformation identification @xcite , finding answers to covid related questions @xcite . in this paper, we present our system in the w-nut 2020 shared task 2: identification of informative covid-19 english tweets @xcite . we use transformer @xcite based models such as bert @xcite to classify covid-19 tweets into informative and uninformative category. we observe that bert heavily relies on death-related information to identify informative tweets, and can be easily fooled by adding a simple death count information to any uninformative tweet. further, to improve robustness of the model, we propose a targeted data augmentation technique. our contributions are as follows: ## method the task is formulated as a binary classification task. since transformers based models provide state of the art performance on the text classification task, we use a pre-trained bert as a classifier. ## experiments for our experiments, we use huggingface's transformers package @xcite . we use adamw optimizer with a learning rate of 4e -5, batch size of 32, and 70 as the max sequence length. we use the pooled representation of the hidden state of the first special token([cls]) as the sentence embedding. a dropout probability of 0.1 is applied to the sentence embedding before passing it to the 1-layer softmax classifier. all models are trained for 7 epochs and the best model is selected on the basis of dev set performance. for the model robustness experiment, we create a modified dev set where we add "10 deaths" to the beginning of all tweets in the dev set and test model performance against that. since the main purpose of this paper is not to identify such universal adversarial triggers @xcite , but to highlight the lack of robustness, our modified dev set provides a proxy for robustness experiment. ## results we use train set to train bert and ct-bert model and report performance on dev set. ## discussion pre-trained language models provide state of the art performance on most of the nlp benchmark datasets. unsurprisingly, ct-bert does well on the shared task dataset. while these models can be further improved using ensemble techniques, improved fine-tuning (?), a more important research direction is to assess the applicability of such models to identify informative information in real-world applications. we show the fragile nature of such classifiers using a simple adversarial task where the model fails to classify an uninformative tweet if we add "10 deaths" to it. while not reported in the paper, we observe similar results for any "n deaths" pattern. on a social media platform such as twitter, fake news or uninformative tweets might contain a similar pattern, and state of the art classifiers might not be able to correctly identify them. it raises serious concerns about the robustness of such systems. while our proposed data augmentation helps in improving the model robustness, it still falls short from what is expected from a trustworthy system. ## conclusion in this paper, we introduce bert based classifiers to identify informative tweets. we show that while such classifiers performs well on standard test benchmarks, they exploits easy clues for classification and their performance degrades drastically in the presence of simple adversarial triggers. we show that targeted data augmentation can help in improving the robustness and classification performance of such classifiers. in future, we will explore universal adversarial triggers @xcite to create a more challenging adversarial dataset and will also explore other techniques such as stability training @xcite to improve model robustness.