Forecasting COVID -19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts (2022)

Abstract: We present a novel approach incorporating transformer-based language models into infectious disease modelling. Text-derived features are quantified by tracking high-density clusters of sentence-level representations of Reddit posts within specific US states’ COVID-19 subreddits. We benchmark these clustered embedding features against features extracted from other high-quality datasets. In a threshold-classification task, we show that they outperform all other feature types at predicting upward trend signals, a significant result for infectious disease modelling in areas where epidemiological data is unreliable. Subsequently, in a time-series forecasting task, we fully utilise the predictive power of the caseload and compare the relative strengths of using different supplementary datasets as covariate feature sets in a transformer-based time-series model.

## introduction many papers have shown that web search data can be used to forecast the spread of infectious diseases @xcite , @xcite , @xcite , @xcite , @xcite . alongside this literature, social media has been exploited for its predictive potential in several other fields such as quantitative finance @xcite , @xcite , logistics forecasting @xcite and election forecasting @xcite , @xcite . research conjoining these two strands has produced results showing that social media can help predict rises in disease caseloads. @xcite and @xcite both used pre-defined keywords in order to predict outbreaks of influenza; words such as "influenza", "fever", "headache" were selected a-priori. these papers assume that useful feature sets have no geographical variation and use the same features regardless of the regional social dynamics; they also assume that useful features are limited to words that refer to symptoms. to address these limitations, @xcite set more general and objective inclusion criteria. for each of four us state covid-19 subreddits, all words over-represented in that us state's covid-19 subreddit compared to the rest of reddit were considered to be potential keywords for forecasting. the most informative keywords proved to be highly dependent on the target state, and included many that did not refer to symptoms. however, the paper still relied on static word counts that miss more complex information as the discussion unfolds over time. the present paper extracts more informative features from social media data and, to our knowledge, is the first work to incorporate modern nlp techniques in this setting. new transformer-based language models @xcite , @xcite , @xcite provide the potential for identifying more informative features for infectious disease forecasting, and using them in a more effective manner. this paper uses transfer learning and clustering algorithms to isolate useful features for predicting covid-19 caseloads. we first pilot-tested a straightforward way to exploit transformer-based language models for the task: the caseload target value was encoded alongside each post in a sequence classification framework. trained using historical data, this approach generates a prediction from every post, and the results are aggregated for an overall prediction. this method performed very poorly because of noise introduced by irrelevant posts, and we do not discuss it further here (see appendix c). to achieve better performance, we developed a novel feature identification technique that filters out unrelated posts and generates informative features using high-density clusters of posts within a subreddit's embedding space. our work builds off @xcite and @xcite who demonstrated that clusters of contextualised word embeddings are a good basis for topic modelling. in a similar vein, aharoni and goldberg (2020) showed that the domain type of a particular text could be identified using the clustering of sentence-level representations. finally, @xcite showed that clusters of contextualised embeddings could detect meaning shifts in words. the success of these papers motivates our use of high-density clusters of sentencelevel representations. the present paper shows that our novel feature sets outperform more traditional methods by comparing our results to those in drinkall and pierrehumbert (2021) in a threshold-classification task. this task provides an understanding of which feature sets provide the most informative trend signals at different caseload growth rates, enabling us to understand the effectiveness of a particular feature type at identifying a distinct epidemiological event. strong performance on this task is relevant in forecasting worst-case scenarios like hospital overflow, where the outcome is a binary variable. the caseload information is not fully utilised in the threshold-classification task. this observation motivates a time-series forecasting task to compare feature sets at predicting a more continuous target. feature selection is a crucial step in time-series modelling @xcite , @xcite ; adding extraneous features to a multivariate prediction can result in performance deterioration as the models get more complex, a fact that inspired l1-regularisation. only highly relevant features, which represent complementary information, improve performance. contributions. we introduce a novel unsupervised method for predicting covid-19 trend signals and forecasting caseloads. we show that sole use of our feature set achieves very high accuracy in trend signal prediction, a significant result for infectious disease modelling in regions where other reported data is unavailable or unreliable. ## datasets comparing our reddit features' performance against other open-source geographically-specific datasets allows us to understand their value. the following data sources were used to create the feature sets in this paper: pushshift api -the pushshift api (baumgartner et al., 2020) is used to compile datasets of target subreddits to create the reddit features. the pushshift api provides data on every comment and submission posted on reddit. this paper uses comments to form the subreddit dataset since there are more comments than submissions, and they constitute more conversational and reactionary discourse. no individual comments or users are reported in this paper to observe the anonymity of the users. update frequency: real-time. ## feature identification social media is a complex and noisy data source, requiring significant processing to isolate meaningful predictive features. the pipeline used in this paper consists of three main steps for feature identification: sentence-level encoding, dimensionality reduction and clustering. this process groups together reddit comments that are semantically similar. following these main steps that are outlined below, the reddit features are reduced further to 25 using a chi-squared test. once these 25 highdensity clusters are identified, the daily counts of comments within these clusters are used as features in the evaluation frameworks in sections 4 & 5. ## threshold-classification framework the threshold-classification framework (henceforth threshold task) uses the same evaluation methodology as in @xcite . the problem is presented as a classification task on balanced classes, with a randomised train/test split and test size of 0.25 on data from 07/03/2020 to 17/01/2021. balanced classes allow us to report accuracy as the performance metric for this task. the feature sets, derived from a 7-day moving average of the datasets in section 2, are concatenated to a target value that encodes whether the caseload increase exceeded the threshold within a given time interval. the threshold is defined by a relative increase, δ r (t): where µ(t) is the 7-day moving average of the caseload, and τ is the prediction horizon. the model used for classification is a random forest (rf) @xcite . the advantage of using an rf model over other tree-based models is that it decorrelates the trees, making it robust to correlated feature sets. social media data is highly correlated as overall take-up surges and wains; therefore, robustness to correlated features is critical. of course, many more complex models would likely outperform an rf model; however, given that the goal of this task is to compare feature sets, the increased transparency that an rf model offers over more complex models justify its use. ## time-series forecasting task this section showcases our feature identification methodology within a time-series forecasting framework (henceforth time-series task) since this is a widely used prediction task in disease modelling. the high-density clusters are used as covariates in two multivariate time-series models. this setup better utilises the caseload feature and learns the temporal patterns within its historical movement. one difference with the threshold task in the feature identification pipeline is the feature pruning step that reduces the number of features to 25. in the threshold task, the target is a binary classification; therefore, a chi-squared test is appropriate. given that the target is continuous in this task, fregression is used. f-regression works by firstly calculating the cross-correlation ρ i of the i th feature x[:, i] and target y: the f-statistic is then calculated along with the associated p-value. then the top 25 most significant features are filtered to make up the feature set. for each model, the training features and targets are normalised between 0 and 1, and the test set is scaled using the same transformation. the target data is changed from the daily caseload to the daily increase in caseload to make sure the time-series is stationary. no moving average is used since the time-series models should account for the weekly seasonality. the models are trained over 50 epochs on data from 07/03/2020 to 31/12/2020 and tested on data from 01/01/2021 to 01/03/2021. whilst it is possible to improve the performance by retraining the model on recently evaluated data and sliding the train-test split across the dataset, our proposed framework highlights how the models perform on completely out-of-sample data. ## discussion examining the contents of our reddit features can help us understand the information they provide to the prediction task. we took the top 5 nonstopwords from the cluster to characterise each cluster and manually named them for better comprehension. as mentioned, the advantage of the threshold task is that it provides greater interpretability than the more black-box time-series models. therefore, the threshold task is used to understand which features are important to the prediction. ## conclusion reddit data performs well at discerning different trend signals for covid-19 caseload increases in the threshold task. reddit features alone achieved high accuracy at most threshold increases but were especially strong when identifying whether the caseload was likely to double in the next 14 days, achieving an accuracy of .970. that value is seen in the time-series task but the performance benefit is not as stark, especially when the number of features increases. the characteristics of reddit data make it appealing: it is readily available and updated in real-time, offering the means for monitoring infectious diseases in regions where reported data is unreliable; however global reddit usage is not constant, and not every area has a subreddit, making our exact methodology hard to scale. as reddit usage increases and disperses around the world or data from another social media site is adapted to fit within our pipeline, the methods used in this paper will become more scalable. another notable conclusion is that the predictive information within reddit data is better extracted by including transformer-based language models in the forecasting pipeline. language model complexity appears to be linked with performance improvements in the threshold task. strong language models allow us to isolate highly specific features predictive of future caseload increases in an unsupervised setting.