T witter Data Augmentation for Monitoring Public Opinion on COVID -19 Intervention Measures (2020)

Abstract: The COVID-19 outbreak is an ongoing worldwide pandemic that was announced as a global health crisis in March 2020. Due to the enormous challenges and high stakes of this pandemic, governments have implemented a wide range of policies aimed at containing the spread of the virus and its negative effect on multiple aspects of our life. Public responses to various intervention measures imposed over time can be explored by analyzing the social media. Due to the shortage of available labeled data for this new and evolving domain, we apply data distillation methodology to labeled datasets from related tasks and a very small manually labeled dataset. Our experimental results show that data distillation outperforms other data augmentation methods on our task.

## introduction due to the worldwide social distancing and lockdown restrictions during the covid-19 pandemic, social media has seen a significant increase in use. large amounts of publicly available twitter data can be used for real-time monitoring of the public agreement/satisfaction or disagreement/frustration with intervention measures during the covid-19 outbreak. this task can be regarded as stance analysis rather than sentiment analysis, because it requires to explore the public opinion on a certain topic rather than just labeling the tweet as literally positive or negative. however, the shortage of labeled twitter data related to covid-19 brings a big challenge for the real-time analysis using supervised learning. therefore, in this work, we focus on the task of automatic data augmentation to allow monitoring the public opinion on covid-19 measures with a minimal investment in time-consuming manual labeling process. our data augmentation approach is inspired by the idea of data distillation @xcite . data distillation is a simple omnisupervised learning method which uses labeled and unlabeled data to update the performance of the model by self-training @xcite @xcite . the previous studies have shown that updated with unseen data in several iterations, data distillation models can reach a consistent improvement. our contributions can be summarized as follows: ## related work during the covid-19 pandemic, a lot of research has been done on twitter data to analyze the public sentiments or responses. to understand the temporal sentiment, @xcite used lexiconbased model to output sentiment score of each tweet. they also explored the public attitude towards specific measures as a case study. in order to analyze the situation information about the epidemic in social networks, @xcite trained a traditional classifier (svm) on manually labeled data, and applied it to label the rest of the data automatically. in our work, we are interested to explore the public opinion on intervention measures taken by the government during covid-19 outbreak, which is a different task from sentiment analysis. the public opinion is defined as stance (support, against) on a target topic, regardless of whether positive or negative language is used in the text. besides, instead of investing in manual labeling of large twitter datasets, we put efforts on developing an automatic data augmentation system, with or without limited amounts of manually labeled data designed for this particular task. data augmentation is critical for supervised learning to tackle the shortage of labeled training data @xcite . to boost the performance of text classification, @xcite proposed to conduct data augmentation using four operations: synonym replacement, random insertion, random swap, and random deletion. their method requires dataset extension with complicated transformations of the original data. @xcite leveraged a large domain-specific corpus to fine-tune a language model for rumor detection, which enabled training data augmentation by weak supervision. in contrast to these works, we aim to explore automatic data augmentation under weak or no supervision, without the need of data transformations. the idea of data distillation has been adopted in various semi-supervised learning scenarios on weakly labeled data. @xcite trained a model on large amounts of labeled data to generate the annotations on unlabeled data, then retrained the model using generated annotations. based on the data distillation framework, the authors of @xcite distilled predictions from a teacher (complex) model to guide the learning of a simpler student model. inspired by the idea of data distillation, we apply it for data augmentation, using two public labeled datasets from related tasks and one small dataset manually labeled by ourselves. ## methodology we adopted distillation methods from @xcite and @xcite to obtain more labeled data. unlike these two dealing with image tasks, we adopted distillation method using language model bert for text analysis task, which is applied to different domain. first, we use a manually labeled dataset to train a basic teacher model. then, we apply the trained model to unlabeled data to get predicted labels. following that, we train a student model which is initialized with identical architecture and parameters as the teacher model, on the union of manually and automatically labeled data. later, we use the trained student model as a new teacher model. then, we iterate the process. during each iteration, only the student model is tested on validation set. finally, we adopt the model, which provides no further performance improvement on validation set. the intuition behind this approach is that a high quality teacher model brings up a good student model, and the improvement of the student model will strengthen the teacher model reversely. the better teacher model is, the more accurate labels will be predicted for unlabeled data, leading to better learning for the student model. on the other hand, during each training iteration, the student model evaluates the reliability of the labels predicted by the teacher model to enhance the teacher model. the pipeline of our data augmentation method is shown in figure 1 . ## datasets we consider three related datasets for our experiments, obtained from two external sources, which are not from the same task or domain. stancedata is the dataset from semeval-2016 task #6 foot_0 for stance detection in tweets. the dataset is about several different topics, unlike our target task containing only one topic. the training set contains 2224 tweets, in which 1395 are labeled as 'against', 753 are labeled as 'favor', and 766 are labeled as 'none'. sentiment140 is the training data foot_1 for sentiment analysis, containing 1,600,000 tweets. 800,000 tweets are annotated as positive, and 800,000 tweets are annotated as negative. as opposed to stance analysis, sentiment analysis only concerns about the literal sentiment, though sentiment features are useful for stance detection @xcite . it is noteworthy that this dataset contains only two classes. lockdowntweets is an unlabelled tweet dataset, which is related to the lockdown policy in new york state of united states during covid-19 pandemic. a large tweet dataset related to the novel coronavirus covid-19 is provided by @xcite . we conducted our work based on the dataset they contributed. we used twitter feature 'lang' to select the tweets written in english. then, we filtered the corpus to obtain only tweets related to new york and lockdown, using a list of keywords. we collected the tweets related to new york using keywords: new york, nys, nyc, governor cuomo. additional keywords were used to select the tweets related to 'lockdown' measure, including: lockdown, stay-at-home, pause, ny on pause, shelter-at-home, nypause, stay home. to monitor agreement or disagreement of public with the 'lockdown' measure, we labeled lockdown-related tweets with 'support', 'against', and 'none' labels. we built annotation guidelines (details are shown in appendix a.1). based on these guidelines, three annotators reached interannotator agreement of cohen's kappa coefficient equal to 0.872 on 100 samples. based on these robust guidelines, two annotators manually labeled a larger set of tweets in this collection by reading the content of each tweet. only tweets assigned the same label by both annotators were used for training and testing. some representative examples from the manually labeled dataset are shown in figure 2 . as result, we obtained a labeled dataset containing 475 tweets with 'against' label, 484 tweets with 'support' label, and 670 'none' tweets. the remaining 29,394 tweets, dated from 22rd january till 10th june, were left unlabeled. we denote the dataset of unlabeled tweets as lockdown-unlabel. the labeled dataset is split into training and test set with the ratio of 2:1, which we call lockdowntrain (326 against, 333 support and 430 none), and lockdown-test (149 against, 151 support and 240 none). ## conclusions and future work in order to monitor public opinion on covid-19 intervention measures from social media in the future, we adapted data distillation method for training data augmentation. the results on covid-19 twitter data show that, using data distillation method outperforms other data augmentation methods. notably, data distillation method on external sources of data shows bigger improvement over transfer learning on the same datasets. moreover, the simple but effective data distillation method with a smaller dataset, manually labeled for our specific task, obtains the best performance. in conclusion, we can recommend to invest in manual labeling of a small dataset and further automatically expand it by data distillation for dealing with a specific task with limited annotation resources. in future work, we will apply this method to obtain more labeled data for more comprehensive public opinion analysis on additional intervention measures, like school closure, wearing face masks, etc, and use it for real-time monitoring of public opinion. the maximum required amount of labeled data will also be explored in future work.