COVID -19 Surveillance through T witter using Self-Supervised and Few Shot Learning (2020)

Abstract: Public health surveillance and tracking virus via social media can be a useful digital tool for contact tracing and preventing the spread of the virus. Nowadays, large volumes of COVID-19 tweets can quickly be processed in real-time to offer information to researchers. Nonetheless, due to the absence of labeled data for COVID-19, the preliminary supervised classifier or semi-supervised self-labeled methods will not handle non-spherical data with adequate accuracy. With the seasonal influenza and novel Coronavirus having many similar symptoms, we propose using few shot learning to fine-tune a semi-supervised model built on unlabeled COVID-19 and previously labeled influenza dataset that can provide in- sights into COVID-19 that have not been investigated. The experimental results show the efficacy of the proposed model with an accuracy of 86%, identification of Covid-19 related discussion using recently collected tweets.

## introduction the typical seasonal influenza virus and the current development of covid-19 have multiple similarities from symptoms to how the virus is spread. both viruses attack the respiratory system, can be spread through asymptomatic carriers, cases can range from mild to severe cases, and are transmitted by contact and/or droplets. influenza and covid-19 both can impact a community negatively due to the contagious nature of the virus and the high number of deaths caused by the viruses. public health surveillance like digital contact tracing @xcite , epidemiological studies @xcite , event detection @xcite , and monitoring the prevalence of vaccinations @xcite can be used to help contain the virus and prevent its spread to the masses. these tools and techniques range from cellphone applications installed personal phones that track the exact spread of a virus @xcite to the development of machine learning-based techniques to study the spread of a virus using social media @xcite @xcite @xcite . similarly, machine learning-based methods have been developed to monitor the public's view on vaccines to combat the anti-vaccine narrative @xcite . using large sources of public information from social media to mine influenza and covid-19 data allows researchers to help gain insight about the viruses. just using a search word like "flu" and "coronavirus" with the twitter api will return millions of tweets with information about vaccines, rumors, symptoms, and family/friends who have contracted the virus. classifying tweets in to smaller subsets including categories like "self vs other" and "awareness vs infection" provides a deeper understanding on how the the influenza and covid-19 are affecting the communities. example tweets of each category can be found in the major road block for using deep learning models on the covid-19 tweets is the lack of annotated data. with millions of tweets related to covid-19 flooding social media, researchers have a difficult time performing supervised learning on the data. we propose a method to attack this problem by transferring knowledge learned in influenza data and integrating it with latent variables obtained from the unlabeled dataset of covid-19 to preform a deeper understanding through self-be able to classify the tweets into related tweets allows researchers to filter out tweets that mention covid-19 in order to attract consumers (click bait). tweets that are classified as awareness is beneficial to researchers when not wanting to look at information regarding users actually becoming infected with the virus. these topics can include perception of masks, rumors of vaccines, etc. @xcite . also provided in the table is a brief explanation of why the specific category is important to understanding multiple aspects of covid-19. these categories allow researchers to study the virus at a more granular level. ## methodology in this section, we explain the set-up of our study and motivate the core components of the proposed covid-19 self-supervised learning with less labeled covid-19 data. we begin with the introduction of our three datasets and data annotation strategy in section 2.1 and follow by describing the core components of our self-supervised learning, as shown in figure 1 . we first learn how to generate the latent representation of unlabeled covid-19 tweets using self-supervised convolutional autoencoder model in section 2.2. subsequently, design our covid-19 supervised downstream task with a pre-trained influnza classification, due to symptom similarities, and fine-tuning of the model using covid-19 few shot learning in section 2.3 and 2.4. we finally present our results and evaluation metrics in section 3 followed by discussion and conclusion in section 4. ## results in ## discussion and conclusion diving further into the data and investigating how the classifiers could be improved, the first step would be to improve the quality of the tweets labels. several tweets in the related category had mislabeled gold labels. for example, "@berniesanders thank you so very much @berniesanders for giving us hope as a nation for and end to this joke of a presidency thank you for all the fund raising and support that you have done for our country in the age of covid19 and good sir" was given the gold label of not related to covid when the model predicted it was. the overall theme was presidential candidate bernie sanders, but it was still related to covid-19. another example, "tell congress to put people first, demand paid sick leave for our most vulnerable workers covid-19". this tweet should be labeled awareness but was given the gold label of infected. this tweet should be labeled as "awareness" but was given the gold label of "infected". with more time to weed out accurate labels and provide higher quality data to the classifier would increase overall accuracy across the board. in future research we aim to use services like amazon mechanical turk to label more data for us. our self-supervised methodology for classifying covid-19 tweets with fewer labeled data has been developed to overcome the challenges of labeling massive covid-19 tweet data. at the time of this research, labeled covid-19 datasets for supervised learning were not readily available. that being said, to achieve maximal results, providing the deep learning models with large amounts of high quality annotations for learning would be ideal. nevertheless, with unlabeled data available and small amount of annotated tweets, our research demonstrates that we can transfer knowledge from unsupervised latent representation and high quality datasets to similar domain classifiers using selfsupervision and few shot learning. lastly, our original hypothesis of influenza tweets and covid-19 tweets being extremely close in context was not entirely true. while covid-19 and influenza tweets have a lot of similarities there also exists differences in the themes of the tweets. twitter users tend to flood their timelines with posts and re-posts of the politics, rumors, misinformation and news related to covid-19 rather than symptoms and infections. looking at the results for each category, leveraging the covid-19 self-supervised pretext training to produce covid-19 latent representations from unlabeled data for supervise downstream covid-19 classifier shows promising results. although the test accuracy of classifier can still be improved, the warm start of covid-19 outperforms the cold start model on 3 out of 4 tasks in terms of the expected accuracy , precision , recall and f1. with this experiment we show that using high quality annotated data in a similar domain, can be used with self-supervision and few shot learning to train classifiers on data where labels are limited. rather than starting from scratch we can initialize the weights on the mlp with the knowledge learned during supervised training. we believe that the architecture and methodology provided in this research show that using self supervision and few-shot learning can overcome some of the challenges of data with limited annotations. using the proposed model to label tweets can assist future researchers to investigate covid-19 tweets at a more granular level.