RollingLDA : A n Update Algorithm of L atent D irichlet A llocation to Construct Consistent Time Series from Textual Data (2021)

Abstract: We propose a rolling version of the Latent Dirichlet Allocation, called RollingLDA. By a sequential approach, it enables the construction of LDA-based time series of topics that are consistent with previous states of LDA models. After an initial modeling, updates can be computed efficiently, allowing for real-time monitoring and detection of events or structural breaks. For this purpose, we propose suitable similarity measures for topics and provide simulation evidence of superiority over other commonly used approaches. The adequacy of the resulting method is illustrated by an application to an example corpus. In particular, we compute the similarity of sequentially obtained topic and word distributions over consecutive time periods. For a representative example corpus consisting of The New York Times articles from 1980 to 2020, we analyze the effect of several tuning parameter choices and we run the RollingLDA method on the full dataset of approximately 4 million articles to demonstrate its feasibility.

## methodological framework the rollinglda method we propose is based on the classical lda @xcite estimated by a collapsed gibbs sampler @xcite and we combine it with the method ldaprototype @xcite , which selects the most reliable lda from a set of models. ## methods we propose the method rollinglda that uses preceding lda results as an initialization for subsequent time intervals. the method builds on an existing implementation of lda @xcite and aims to ensure consistent time series based on textual data. the method provides a memory parameter to use a different number of time units of the past as initialization to find a good trade-off of consistency and flexibility of topics. different values for the memory parameter can be investigated quantifying topic-self-similarities over time. the method is implemented and published as r package rollinglda @xcite and its source code can be retrieved at https://github.com/ jonasrieger/rollinglda . ## stability and sensitivity analysis for a brief demonstration of which of the presented similarity measures is particularly well suited for the present case of comparing topics at different points in time, we use zipf's law @xcite . this states that for an ordered list of v entries, such as words in this example, the relative frequency of the element with rank r can be written as we consider how stable the similarity measures are in the uncertainty scenario and how sensitive they are to detect strong changes in the topics. ## analysis in the following, the proposed method rollinglda is applied to an example dataset. the calculations were performed using r (r core team, 2021). figure 3 : unit-to-unit self-similarities of the topic health in the scenario of quarterly modeling with three quarters memory and starting with the rolling approach in 1985 for the six different similarity measures. ## discussion we presented a method rollinglda to model consistent time series from textual data, which is also suitable for monitoring applications due to its efficiency. in particular, it is possible to choose very frequent update intervals and thus to keep the runtime of each update very short. apparently, the specific parameterization is not that important, the model seems relatively robust. it is less sensitive with respect to its parameter choice, so that even for more inappropriate parameter choices, the model produces plausible results. our study has shown, for example, that there is no strong difference between changing previous assignments and fixing previous assignments. however, the latter has a considerable runtime advantage, because the gibbs sampler does not have to iterate over the previous assignments (the memory) in each time step. for runtime reasons, we therefore recommend the version with non-changing previous assignments. we also recommend to choose the memory parameter reasonably. it is an important and intuitive parameter, which specifies how much (modeled) past the model takes into account for modeling the next chunk. for example, three quarters of memory in a quarterly modeling scenario means the consideration of one year for each modeling step. when choosing this parameter, one should consider seasonalities, because a topic that only appears in summer, for example, could disappear repeatedly due to a memory that only lasts for one quarter. in case of reappearance it is then not ensured that it receives the same index. instead, it joins the most similar topic, so that the coherent interpretation of the topic can not be guaranteed. in addition, the initial lda should cover a time horizon as short as reasonable, so that a large part of the time series is covered by the rolling approach and can be interpreted accordingly. we also tested sequential prototypes instead of sequential ldas (cf. line 9 in algorithm 1). however, it turned out that the set of possible ldas is very similar such that we observed no further practical gain using the ldaprototype for each sequential lda step. further research could include weighting the previous documents for the memory or looking at a random sample of those. for the latter case, the consideration of reliable methods for the determination of the update states then again could be interesting. in the long term, one goal is to extend the method to varying numbers of topics per time interval.