Analyzing social media using topic models has become popular for studying and tracking various public health events around the world [35, 11, 34] . Topic modeling is an unsupervised machine learning technique used to reveal latent themes from large text datasets. Dynamic topic modeling investigates how topics evolve in a sequentially organized corpus of documents, where the data is typically divided by time slices e.g. by day [38, 5, 19, 18] . Here, we perform dynamic topic modeling in two ways. The first investigates the change in prevalence of topics over time while the second considers the evolution of the topics themselves over time. A popular topic modeling method is nonnegative matrix factorization (NMF) [50, 26] . Natural extensions of NMF that capture topic evolution are its online versions, when the data can be updated with time, or tensor analogues [1, 43, 10, 2, 47] . Here, we analyze the use of NMF and tensor nonnegative CP decomposition (NCPD) for dynamic topic modeling. Other popular dynamic topic modeling techniques include latent Dirichlet allocation [20, 54] , a probabilistic approach as opposed to the dimension reduction approach of NMF and NCPD.
Here, we consider data from the the social media platform, Twitter. During the COVID-19 pandemic, Twitter has experienced increased usage including discussion and dissemination of information relating to the pandemic [44, 9] . A number of related works consider Twitter and other social-media data related to the COVID-19 pandemic via various statistical and learning approaches, with specific aims to understand the effects and prevalence of bots and misinformation [14, 51] , polarization [15, 6] , sentiment and emotional state [52, 22] , gender differences [46] , racism and xenophobia [55] , and other aspects [33] . In this work, we illustrate the abilities of four different unsupervised nonnegative matrix and tensor factorization approaches for dynamic topic modeling to study the topics learned from social media and how they coincide with events portrayed in news outlets. We compare the various advantages of the methods and how they incorporate temporal information, and showcase that the topics identified may be used to gauge how discussions among the public change over time, even in tumultuous circumstances.
1.1. Data. We consider Twitter text data related to the COVID-19 pandemic from Feb. 1 to May 1 of 2020 [9] . Twitter is a social media website where users post and interact with short texts known as tweets. The number of COVID-19 related tweets is large, typically containing hundreds of thousands of tweets each hour. While the online methods considered are well-suited to handling large streaming data, we subsample this dataset for ease of computation and to compare results on a fixed dataset with more standard methods that require loading the data into memory. Specifically, we consider the top 1000 retweeted English tweets from each day. Using the most retweeted tweets, limits the data to information that was shared and spread widely.
Tensors are common algebraic representations for multidimensional arrays. The order of a tensor is the number of dimensions, which is also referred to as ways or modes [23] . We denote vectors with lowercase letters x with x(k) denoting its k th entry, matrices with uppercase boldface letters, X, and thirdorder tensors with uppercase calligraphic letters X . For a matrix X, the vector x k denotes its k th column. We let · F and · 1 denote the entrywise Frobenius norm, and the entrywise L 1 norm respectively. The set of nonnegative real numbers [0, ∞) is denoted R + . We let ⊗ denote the outer product of two vectors. For tensors A and B of the same size, denote by A B the Hadamard (pointwise) product. When B is a matrix, for each 1 ≤ j ≤ n, we denote their j -mode product by A × j B. See [23] for an excellent survey of related definitions and tensor algorithms.
1.3. Contributions. We analyze Twitter text data related to the COVID-19 pandemic [9] using four unsupervised topic modeling techniques, specifically NMF and NCPD as well as online variants that are well-suited for large data. We illustrate each of these methods as a dynamic topic modeling technique, revealing latent themes in the data with emphasis on temporal evolution.
From the text data, the methods discover topics trending in news sources, including political events, personal beliefs about COVID-19 and calls to action, and successfully attribute them to the days they were trending (see Table 1 for a rough timeline and sources used 1 ). We find that tensor methods capture more topics that are present primarily over short time periods compared with NMF and online NMF. We discover many such fleeting topics in the early days of COVID-19, while larger trends dominate as time goes on. We demonstrate that online counterparts to NMF and NCPD produce similar results and are viable alternatives that allow for processing much larger amounts of data without subsampling. To our knowledge, this is the first application of online NCPD to text data. We compare and contrast the various approaches, showing there are advantages to using each method, and that significant pandemic related information can be obtained through monitoring social media data.
We describe how the matrix decompositions NMF and online NMF can be used to analyze the dynamics of topics discussed on Twitter during the first several months of the COVID-19 pandemic.
2.1. Nonnegative Matrix Factorization. NMF is a popular tool for extracting hidden topics from text data [7, 25] . For a data matrix X ∈ R m×n , one learns a low-rank dictionary W ∈ R m×r and code matrix H ∈ R r ×n that minimize X − WH 2 F , where r > 0 is typically chosen such that r < min{m, n}. The dictionary matrix W represents topics in terms of the original features (in our case unigrams and bigrams). Each column of the code matrix H represents a data point as a linear combination of the dictionary elements with nonnegative coefficients. We use NMF to learn a dictionary W from all data and analyze topic dynamics through changes in topic prevalence over time in the code matrices H i from each day's data for a total of T = 90 days.
2.2. Online NMF. Similar to NMF, we use online NMF (Algorithm 1) to learn global topics and then determine the distribution over topics for each time interval. In online NMF, however, the topics are learned sequentially from the data, allowing past learned topics to influence those learned for current data [28, 30, 16, 53] .
Online NMF has some advantages over standard NMF when applied to large temporal data. First, online NMF can be used with streaming data as opposed to processing all of the data at once. This ability is especially useful if data is continually added or if the data is too large to load into memory at once. Second, the dictionary W can be updated to reflect historic topics, but also incorporate changes in the data. The balance between preserving historic topics and updating the topics based on new data is controlled by the learning rate. The learning rate, t −β , decreases as the number of updates t increases (see Algorithm 1) . Smaller values of β ≥ 0 lead to a larger learning rate, so that changes in the most recent data have a stronger impact on the learned topics.
When learning topics, online NMF only requires storing the current dictionary matrix W, data from a single time slice, X t , and the current code matrix H. If loading a single time slice, X t , requires too much memory, a subsampled X t can be loaded into memory at each inner iteration. On the other hand, NMF requires access to the full data matrix at each iteration [30] . See Table 2 for a comparison of the memory requirements of the methods considered.
Under certain conditions, the expected loss for the decomposition learned by online NMF converges almost surely to that of NMF [28, 30, 16, 53] . Despite the fact that the streamed COVID-19 related tweets do not exhibit the 1 The topics learned from data and news references may contain factually inaccurate information. While we include some news references to validate results, many additional sources exist. We make no claims that these sources are factually accurate.
Require: data slices X t , initial matrix W, learning rate parameter β 1: Setf 0 ≡ 0 2: for t = 1, . . . , T do Calculate dictionary W 3: for i = 0, . . . , iterations do 4: X t ← subsample minibatch from X t
H ← argmin H∈R r ×n 
We describe and compare the use of NCPD and online NCPD for analyzing the dynamics of topics of the COVID-19 Twitter dataset organized into a 3-order tensor, where the modes correspond to time (in days), vocabulary (alphabetical), and tweets (descending in frequency).
A main goal of dynamic topic modeling is to extract latent themes in the data along with their temporal evolution. One strategy for achieving this is to decompose higher-dimensional data tensors into interpretable lower-dimensional representations. Similar to NMF, one might adapt tensor decompositions with some additional structure, such as nonnegativity, which allows for interpretability of topics. NCPD is considered as a dynamic topic modeling technique for tensor data that successfully showcases topic variation across all modes of the tensor (including the temporal mode), thus preserving the temporal information in the data [1, 10, 2] .
NCPD factorizes a tensor into a sum of nonnegative component rank-one tensors, defined as outer products of nonnegative vectors [8, 17] . In particular, given a third-order tensor X ∈ R n 1 ×n 2 ×n 3 + and a fixed integer r > 0, the approximate NCPD of X seeks matrices A ∈ R n 1 ×r + , B ∈ R n 2 ×r + , C ∈ R n 3 ×r + , such that X ≈ r k=1 a k ⊗ b k ⊗ c k , where the nonnegative vectors a k , b k , and c k are the columns of A, B, and C, respectively. The matrices A, B, and C are referred to as the NCPD factor matrices. To obtain a nonnegative approximation with fixed rank r we consider minimizing the reconstruction error, X − r k=1 a k ⊗ b k ⊗ c k F among all the nonnegative vectors a k , b k , and c k . Several approaches have been proposed for solving this problem including, alternating least squares method, proposed in the original papers [8, 17] , and the multiplicative update algorithm [39] .
Applying NCPD to the third-order tensor data X , we obtain three factor matrices capturing topic variation across each mode: time representation of topics (changing topic prevalence through time: emerging, trending, fading etc), term representation (the unigrams/bigrams that characterize each topic) and tweet representation (the tweets associated with each topic).
We consider a special case of the recently introduced online CP-dictionary learning [41] as an online counterpart of NCPD. We specifically assume that the factor matrices are constrained to be nonnegative and call the method online NCPD (Algorithm 2).
Given a probability distribution π on the set of data tensors R n 1 ×n 2 ×n 3 + , online NCPD seeks nonnegative factor matrices A ∈ R 
where the random data tensor X is sampled from the distribution π and λ ≥ 0 is a sparsity regularizer. For each realization of X , the optimal choice of h ∈ R r ×1 + gives the nonnegative coefficients to combine the rank-1 tensors
One can think of such h's as the columns of the code matrix H in NMF. Minimizing Equation (1) is equivalent to minimizing the NCPD reconstruction error when the distribution π is supported on a single data tensor and λ = 0. In this case, one can absorb h(k) into a k as there is a single data tensor to factorize.
The expected reconstruction error f (Equation (1)) is approximated by the empirical reconstruction error
where (X , A, B, C) denotes the infimum in Equation (1). The theory of empirical processes [45] guarantees f − f t ∞ → 0 almost surely as t → ∞ when the factor matrices A, B, C are restricted in a compact set and the data tensors X s are asymptotically distributed as π with some mild condition (see also [28, Lem. 7.10] ). Hence the problem of minimizing the expected reconstruction error f in Equation (1) can be reformulated as follows: Data tensors X t arrive online (sequentially), and for each t ≥ 1, one finds factor matrices A t , B t , C t that minimize the empirical reconstruction error f t in Equation (2). However, this is a computationally difficult problem since f t is nonconvex. Algorithm 2 gives a high-level description of online NCPD. Given the new observed tensor X t , we first find its best linear rank-1 approximation using the current best factor matrices A, B, C, then update a surrogate loss functionf t that majorizes the empirical loss function f t in Equation (2), and then update the factor matrices A, B, C sequentially with the updated surrogate loss function. This follows the general scheme of Stochastic Majorization-Minimization [29] , which also generalizes the online NMF algorithm in Algorithm 1. Defining the full surrogate loss functionf t as in Algorithm 2 (and hence storing all previous data tensors X 1 , · · · , X N in memory) can be avoided by using aggregate tensors of size R r ×r + and R n 1 ×n 2 ×n 3 ×r +
. See [41] for implementation details. The coding step (Line 3) of Algorithm 2 is a convex problem and can be easily solved by a number of known algorithms (e.g., LARS [13] , LASSO [42] , and feature-sign search [27] ). The surrogate loss functionf t is quadratic in each matrix coordinate, so each of the three subproblems in the factor matrix update step in Algorithm 2 is a constrained quadratic problem and can be solved by projected gradient descent (see [30, 28, 41] ). The more general version of online NCPD was shown to converge almost surely to the set of stationary points of the expected reconstruction error in Equation (1) under mild regularity assumptions of the sub-problems and Markovian dependence in data tensors X 1 , · · · , X N [41] .
We apply NMF, online NMF, NCPD, and online NCPD to recover topics related to the COVID-19 pandemic discussed on Twitter and their prevalence over time. The dataset is detailed in Subsection 1.1 [9] . Note that the objective functions for NMF and NCPD are nonconvex and, thus, the resulting low-dimensional representations depend on the optimization algorithm used and its initialization. Twenty topics are used in order to recover both general topics that occur over longer time spans (for example, new cases, social distancing) and topics that are discussed over shorter time periods (such as specific outbreaks, cruise ships). Table 1 gives a timeline with notable events discovered in the data.
News 2 Feb 'No Meat, No Coronavirus' [49] 3 Feb COVID-19 cruise ship outbreak [21] 7 Feb Death of Dr. Li Wenliang [3] 8 Feb COVID-19 death toll overtakes SARS [12] 18 Feb Spike of cases in South Korea [40] 26 Feb Mike Pence appointed to lead Coronavirus task force [37] 28 Feb 'Trump calls Coronavirus Democrats' 'new hoax" [31] 29 Feb First COVID-19 death in the U.S. [32] 11 Mar WHO declares a pandemic [48] [36] . Unigrams and bigrams are considered and referred to as terms. We use the default tokenizer and parameters, and limit the vocabulary size to 5000. The NLTK English stopword list [4] is removed as well as non-word sequences such as "https". We additionally remove words that are essentially synonymous with COVID-19 as all the tweets in the dataset are related to this common topic. Specifically, we remove all words containing "coron" or "cov" that are not followed by the letter "e". is represented by a positive linear combination of terms. Terms with larger values being more significant to the topic.
We present results for each of the methods in the form of a heatmap that summarizes both the term representation and temporal prevalence of topics (see Figures 1 to 4) . For NMF and online NMF, the mean topic representation for each day is given in the columns of the heatmaps, while for NCPD and online NCPD, the factor matrix showcasing the temporal representation of the topics is shown. Each row of the heatmaps corresponds to a learned topic and a three-term summary of each topic is included for convenience 2 . Further, each column of the heatmap indicates the distribution over the extracted topics for each day. For online NMF and online NCPD, we also present three-month evolution of r = 5 topics and their prevalence (see Subsection 4.7).
Each of the methods separately recover several common trends in the data. Generally, China-related topics are most prominent in early and mid Feb. The prevalence of these topics then decreases in mid Feb. A topic relating to new cases spikes in prevalence in mid Feb. as outbreaks begin to occur around the world. In late Feb. to mid Mar., a topic relating to U.S. President Trump and his administration's response spikes in prevalence. Separate "social distancing", "stay home," and "lockdown" topics begin in early to mid Mar. These topics typically persist throughout Apr.
We first consider standard NMF and plot the average distribution over learned topics for each day in Feb.-Apr. of 2020. Results are shown in Figure 1 (see Subsection 4.1 for plot description). The learned topic prevalence is largely as expected. China related topics 5 and 12 are most prominent in early and mid Feb. The prevalence of these topics then decreases in mid Feb. Topic 7, relating to new cases and deaths, peaks in mid Feb. as the virus spread around the world, particularly in Italy and Iran. In late Feb., the topic relating to U.S. President Trump and his administration's response spikes in prevalence as the topics relating to China decrease in prevalence. An "outbreak/spread" topic is gradually replaced by a "pandemic/global pandemic" topic in early Mar., which coincides with the World Health Organization's declaration of a pandemic on Mar. 11 [48] . One of the most prevalent topics from late Feb. on is a general urgency topic (topic 4: "get", "need", "time"). 2 The process for generating term summaries is detailed in the appendix. Keywords and their associated weights for the learned topics are also summarized in tables in the appendix. Figure 2 , with learning parameter β = 0.7, minibatch size of 50 tweets, and 100 inner iterations. The progression of the learned topics with the same parameters but five topics is shown in Figure 5 . Comparing The similarity of the results for NMF and online NMF suggests that despite no guarantee of convergence for this temporal data online NMF is a reasonable alternative to NMF for streaming data and for data that is prohibitively large for NMF. The topic prevalences are, however, more evenly spread among the topics, especially from mid Mar. through Apr. Some differences between NMF and online NMF are the inclusion of topics relating to New York (topic 5), India (topic 19), the UK (topic 11), and a "health workers" topic (topic 18) in online NMF. Topics relating to Japan, Hong Kong, cruise ships, and the CDC appear with NMF, but do not occur in the learned online NMF topics.
We compute the rank-20 NCPD of the tensor data X ∈ R 90×5000×1000 + with multiplicative updates [39] using TensorLy [24] . Figure 3 shows the normalized factor matrix representing the distribution over the topics for each day. We omit the factor matrices for the term representation, and tweet representation of topics due to their high dimensionality. We instead showcase topic summaries (see the appendix for details). In Figure 3 , we primarily observe two different types of temporal behavior among the topics: (i) topics that persist through a period of time, and (ii) topics that are short-lived (i.e. prominent for 1-3 days). The topics that persist through a period of time, admit various temporal behaviors. Topics 5 and 11 are persistent, and primarily capture China and Wuhan related COVID-19 events. The topic summaries indicate that topic 11 captures "new cases" in Wuhan and more generally China, whereas topic 5 captures the events of Pakistan standing with China that peaks on Feb. 8 and then decreases in prevalence. We observe that topics 6 and 7 primarily capture lockdown and social distancing events, where in addition topic 7 incorporates the Easter holiday, and topic 6 "government/trump" events. Both topics primarily trend in Apr., with topic 7 peaking on Sunday, Apr. 12 (Easter), and topic 6 increasing in prevalence in the end of Mar. In addition, we observe that topics 13 and 14 primarily capture a "pandemic" topic, with topic 13 relating to deaths, whereas topic 14 relates to new cases. Indeed, we observe in Figure 3 that topic 13 (deaths) gradually replaces topic 14 (cases). Furthermore, a general stay home, urgency topic ("please", "stay safe", "help"), evolves and experiences a significant peak in mid Mar. On the other hand, certain topics admit a sparse temporal representation, which indicates that they were present for only a short period of time. The timing of these learned topics aligns with related events as can be seen in Table 1 . For example, we observe in Feb. that topic 2 is followed by topics 18, 19, 10, and 12 in Figure 3 . 4.6. Online NCPD. For online NCPD (Algorithm 2), the input is a sequence of tensors (X i ) 1≤i ≤50 ∈ R 90×5000×100 + generated by choosing 100 slices independently and uniformly from the tweets mode of the data tensor X . Rank r = 20, sparsity regularizer λ = 1, and learning rate β = 0.7 are used. The resulting factor matrices for topics and their temporal evolution are shown in Figure 4 (see also the timeline in Table 1 ). Topics that are present in early-mid Feb. that decrease in prevalence toward the end of Feb. include topics 1, 7, 8 for the outbreak of the virus in Wuhan, topic 4 for the cruise ship passengers in Japan, and topic 10 for the outbreak in Iran, South Korea, and Japan. Other topics emerge in late Feb. to early Mar. and decrease in prevalence toward Apr., such as topics 11 and 13 for staying home and social distancing, topic 14 on political terms such as U.S. President Trump, hoax, and Mike Pence. Lastly, topics such as topics 12, 16, and 20 on lockdown and pandemic become more prevalent towards the end of Apr. Online NCDP also detects topics that are sharply concentrated in a short period of time (1-3 days) such as beliefs surrounding eating meat and COVID-19 (topic 6) Pakistan standing with China (topic 9), topic 5 on the outbreak in Italy which peaked around Feb. 22. Some short-lived topics detected by NCPD in Figure 3 are not captured by online NCPD, such as Vice President Mike Pence's appointment as chair of the White House Coronavirus Task Force, the first COVID-19 death in Washington, and the death of Dr. Li. 4.7. Sequential Learning with Online Algorithms. Online NMF and online NCPD can be used to detect how topics themselves, not only their prevalence, evolve over time. To do this, we partition the data into a sequence of inputs. For online NMF, the topic matrices are continuously adapted to data from each day by Algorithm 1 (with β = 0.7, minibatches of 50 tweets, and 100 iterations). Topic prevalence within each month is computed with the topic matrix learned at the end of the month. For online NCPD, each tensor X s ∈ R 30×5000×1000 + , s = 1, 2, 3 contains a month's data. For online NCPD, topics and their prevalences are learned simultaneously from each monthly slice X s using Algorithm 2 (subsampling (30, 5000, 100)-tensors from X s for N = 100 iterations with β = 0.7 and λ = 0.1), where the prior factor matrices are taken to be the initial factor matrices for the algorithm freshly applied to the next slice. In Figures 5 and 6 , topics are represented as wordclouds, and colorbars show the prevalence of the topics for each day of the month chronologically. The topics learned in Feb. primarily focus on China, Wuhan, South Korea, Hong Kong, and Trump, while by Apr., the topics move away from discussing specific places and focus on general topics such as staying home, new cases and deaths, testing, lockdowns and social distancing. Further, some short-lived topics that occur in a particular month are detected in Figure 6 , but not elsewhere. For example, topic 2 of Figure 6 in Mar. captures the historic plunge in oil price and topic 1 in Apr. relates to the the World Health Organization.
Lastly, we compare topic prevalences in each month in Figures 5 and 6 . When using online NCPD, the topics and their prevalences are learned at the same time, and the following month's topic prevalence is updated from the previous month's. Hence is not likely to have both the topic content and prevalence change drastically at the same time. For example, topic 2 in Figure 6 sharply peaks at around day 5-7 in each month, where topics adapted significantly from Dr. Li to oil to staying home. On the other hand, when using online NMF, one first learns the topic representations from each month's data and then prevalences of those topics throughout the month are computed. Consequently, while the topics typically change gradually each month due to the continuous learning, this is not necessarily true for the the post-processed prevalences, as we observe in topic 3 of Figure 5 4.8. Method Comparison. We find that NCPD and online NCPD are better at learning and identifying topics that occur over short time periods. On the other hand, NMF and online NMF primarily detect topics that occur over longer periods of time (at least a week). Thus, for a given day, the focus is typically spread over a larger number of topics instead of having topics that occur as short intense pulses.
A major advantage of the online methods is reduced memory requirements as the data is processed in slices. On the other hand NMF and NCPD hold the entire dataset in memory during the learning process. See Table 2 for a summary of memory requirements. 
We demonstrate NMF, online NMF, NCPD, and online NCPD as dynamic topic modeling techniques that are able to identify significant topics discussed on Twitter relating to the COVID-19 pandemic and detect changes in the prevalence of these topics over time. We validate our results using outside news sources, and show that the methods discover topics trending in news sources, and successfully attribute them to the days they were trending (see Table 1 ). We discuss and compare the temporal topic patterns learned through each of these methods. For large data, online variants, online NMF and online NCPD serve as viable alternatives for learning topics and their temporal patterns. NCPD and online NCPD appear to better identify topics that experience a lot of interest over only a few days compared to NMF and online NMF. All methods considered detect major topics that are present over multiple weeks. Future work includes studying how extracted topics vary when considering various sampling schemes, and comparing the extracted topics from Twitter text data with other social media platforms.
We thank Jacob Moorman for his contributions to the code used. Molitor, Needell and Rebrova were partially supported by NSF CAREER DMS #1348721 and NSF BIGDATA #1740325. Elizaveta Rebrova also acknowledges sponsorship by Capital Fund Management.
We report summaries of each topic based on the top keywords associated with each topic for NMF, online NMF, NCPD, and online NCPD in Tables 3 to 6 . Each learned topic is represented by a positive linear combination of terms. Terms with larger values in a particular topic are more significant for that topic and, thus, the terms with the largest values provide interpretable descriptions of the topics. Frequently, a bigram and one or both its components occurs as the top weighted terms for a topic (for example, ("stay", "safe", "stay safe")). To derive more concise topic summaries, we use the terms with the largest values for each topic, but exclude unigrams that are included in a bigram whose weight is at least half that of the unigram. We additionally exclude terms that do not contain any alphabet characters. 