Statistically Evaluating Social Media Sentiment Trends towards COVID -19 Non-Pharmaceutical Interventions with Event Studies (2021)

Abstract: In the midst of a global pandemic, understanding the public’s opinion of their government’s policy-level, non-pharmaceutical interventions (NPIs) is a crucial component of the health-policy-making process. Prior work on CoViD-19 NPI sentiment analysis by the epidemiological community has proceeded without a method for properly attributing sentiment changes to events, an ability to distinguish the influence of various events across time, a coherent model for predicting the public’s opinion of future events of the same sort, nor even a means of conducting significance tests. We argue here that this urgently needed evaluation method does already exist. In the financial sector, event studies of the fluctuations in a publicly traded company’s stock price are commonplace for determining the effects of earnings announcements, product placements, etc. The same method is suitable for analysing temporal sentiment variation in the light of policy-level NPIs. We provide a case study of Twitter sentiment towards policy-level NPIs in Canada. Our results confirm a generally positive connection between the announcements of NPIs and Twitter sentiment, and we document a promising correlation between the results of this study and a public-health survey of popular compliance with NPIs.

## introduction as covid-19 spreads rapidly around the world, governments have implemented different npis to contain the spread of the virus. while effective at slowing down the spread of covid-19 @xcite , npis such as school and non-essential businesses closures, telecommuting, mask requirements and physical distancing measures have drastically changed our lives and sparked dissent. antimask and anti-lockdown protests are commonplace, while there are nearly fifty million active cases around the world. it is crucial for decision makers to understand the public's opinion about npis, and for policy-makers to have a means of forecasting the level of popular compliance with them. this will determine their effectiveness as well as whether additional measures and communication strategies are needed in light of waning adherence. analysis of social media data is already popular among epidemiologists, as it is a data source with near real-time feedback at very low cost @xcite . extracting sentiment trends towards the pandemic on various social media platforms has already attracted interest @xcite @xcite . neural sentiment analysis is very prevalent because of its high performance on classification tasks foot_0 and versatility. temporal variation of sentiment is usually represented by time series, in which an average model-predicted sentiment scores over from all social media posts within each time interval is computed. previous work following this paradigm suffers from two major issues, however. firstly, nearly all time-series analyses have been based on sentiment classification results -every post is classified into one of the predetermined sentiment categories (positive/(neutral)/negative) -even though sentiment is a continuous random variable. for example, @xcite provide two "sentiment-neutral" examples that in fact have differing sentiments. smoothing sentiment from a continuous variable into a ternary or binary scale causes a loss of dynamics, hence increasing the difficulty of the task and lowering the reliability of all subsequent analyses. there are now n-valued sentiment corpora for @xmath0, but finer-grained discrete sentiment does not entirely solve the problem. the valence regression task (v-reg) proposed by @xcite is far more suitable because it conveys a continuous sentiment intensity measure through a logistic regression score. a continuous score also allows us to compute an average sample sentiment over a definite period of time, which has a more accurate variance than smoothing binary scores. secondly, because of the community's lack of a model capable of conducting significance tests and distinguishing the influence of various events across time, no statistically sound conclusion can be drawn. as an example, @xcite claimed to have noticed a link between public sentiment and the timing of the dutch government's press conferences by visually inspecting the raw trend of social media sentiment, seen in figure 1 . in fact, there were numerous peaks and valleys throughout the interval they studied, because the average sentiment fluctuated wildly during this time. we can bring the potential of this urgently needed application to fruition by looking outside cl/nlp. financial analysts face similar problems when they try to assess the effect of a particular news event on the price of a particular stock, because the price is affected by countless events as well as the reactions of traders with different motivations and perspectives on those events. event studies @xcite have been proposed and recognised as viable methods for attributing stock price fluctuations to specific financial events. to our knowledge, there has been no study of this class of methods within epidemiology. ## experimental setup gilbert et al. @xcite started collecting covid-19 related tweets by searching for tweets mentioning at least one of the various naming conventions for covid-19 using the twitter search api as at january 21, 2020, and collected 281,487,148 tweets up until august 23rd, 2021. after carmen geolocation @xcite , we obtained 5,979,759 english twitter samples from canada. for this paper, we studied two npis: wearing a mask and social distancing. for present purposes, we considered an event to be every change in the stringency level of any npi, as measured by the oxford covid-19 government response tracker (oxcgrt) project @xcite . we used a keyword-based filter to obtain topic-related tweets. we began with a manually written list of related keywords to obtain a list of tweets m that contain a keyword, and m that do not contain any keyword. then for each bigram and trigram x, we calculated a topic relevance score based on pointwise mutual information: pmi(x; m )-pmi(x; m ). we ranked the top 150 keywords for each n-gram and manually removed the topic-unrelated ones. for example, "covidsafe" was identified using this method but "congressman sponsor," a topic relevance score of 14.59, was nevertheless manually removed. after filtering all the tweets connected to an npi of interest, we computed their valence score using the ntua-slp model, 2 which was selected from the 75 entrants to the v-reg shared task @xcite . we followed the hyperparameter settings from the original paper @xcite and reproduced its reported pearson correlation (0.846) on the english valence dataset. to establish a periodic time series of valence change, we computed the daily average valence of tweets posted on the same day. foot_2 ## individual npis experimental results wearing a mask canada's mask advisory has changed several times during the progression of the pandemic @xcite and we investigated two key changing points of the advisory as events. on april 6th, 2020, the public health agency of canada (phac) revised the advisory for mask wearing (event 1), permitting the use of non-medical face coverings in public @xcite . finally on may 20, 2020, phac formally issued a recommendation for the general public to wear masks in public (event 2) @xcite . assuming a confidence threshold of α = 0.05, event 1 had a statistically significant positive impact for up to 9 days (figure 2b ). event 2 also showed significance from two days after the event to up to eight days after ([+2, +8]; figure 2c ). unlike event 1, there is also a period of significance right before the event occurred. this may have been anticipatory, or it may indicate that the observed impact had instead been caused by prior events. during the 9-day effect window of event 1, there is a 2.13% positive car, with t-statistic 1.73, wilcoxon statistic 7.0, and z-statistic 1.67. social distancing social distancing recommendations have been issued with different stringencies and at different times at the provincial level in canada. therefore, we focus separately on three provinces: ontario (on), british columbia (bc) and alberta (ab), with sufficient numbers of tweets and different distancing policies. according to @xcite , ontario released its first provincewide social distancing recommendation on march 16, 2020 @xcite ; british columbia issued a social distancing recommendation on march 17, 2020 @xcite ; and lastly, alberta released a public message about social distancing on march 21st 4 (mccoy et al., 2020). figure 3 analyses the significance of the initial recommendations in those three provinces. all three announcements have a positive impact on car with statistical significance. ontario's recommendation (figure 3d ) has a short but significant impact on [+2, +7]. alberta (figure 3f ) exhibits a significant impact on [+3, +9], and british columbia on [+1, +9]. ## car and survey data correlation to help understand whether the sentiment of npis measured using twitter are representative of the general canadian population, we assessed the correlation between our npi sentiments and the level of compliance measured through a national survey. the covid-19 monitor initiative @xcite has conducted 25 surveys in canada on people's compliance with 6 npis since mid-march. each survey has approximately 2000 participants. the demographics of the participants have been pre-stratified, and each wave was post-stratified by modelling raking weights based on the 2010 canadian census. among the 6 npis, both social distancing and wearing a mask appear. for the cross-correlation test, both time series have been detrended using the scipy signal package, foot_4 and then pre-whitened following the instructions proposed by @xcite to remove autocorrelations with the time series. 6 figure 4 shows the correlations and crosscorrelations with the proportion of the population who report complying with either of these two npis and car. wearing a mask receives a strong pearson @xmath1.915 (figure 4a ), a cross-correlation of 0.710 and a +5 lag, meaning car is 5 days ahead of the survey (figure 4b ). social distancing receives a moderate pearson @xmath2.481 (figure 4c ), a cross-correlation of 0.492 and also a +5 lag (figure 4d ). the cross-correlations cannot be quantitatively compared with the pearson correlation scores as they are calculated differently, but the general trend stays the same: wearing a mask exhibits a strong correlation while social distancing, only moderate one. the lags also accord with our expectations as cov @xcite conducted surveys 4 to 10 days apart. the lower correlation for social distancing might have been caused by their more diverse implementation across subsovereign jurisdictions (see section 4). as the details of the sample selection process at the provincial level are not publicly available, we have not been able to draw direct, provincial comparisons. mask-wearing advisories, however, are mostly issued at the federal level in canada. comparing mask-wearing across provinces is thus less problematic. with both types of npi, twitter users are demographically younger, better educated, and more urban than the general population @xcite . this may explain some differences from the national distribution sampled for this survey.