Evaluating Dynamic Topic Models (2024)

Abstract: There is a lack of quantitative measures to evaluate the progression of topics through time in dynamic topic models (DTMs). Filling this gap, we propose a novel evaluation measure for DTMs that analyzes the changes in the quality of each topic over time. Additionally, we propose an extension combining topic quality with the model’s temporal consistency. We demonstrate the utility of the proposed measure by applying it to synthetic data and data from existing DTMs, including DTMs from large language models (LLMs). We also show that the proposed measure correlates well with human judgment. Our findings may help in identifying changing topics, evaluating different DTMs and LLMs, and guiding future research in this area.

## introduction dynamic topic models (dtms) @xcite learn topics and their evolution over time from a time-indexed collection of documents. variants of dtms include traditional statistical topic models, neural vae-based topic models, and topics learned using large language models (llms). dtms have proven useful in various domains, including text mining @xcite @xcite , computer vision @xcite @xcite , and computational biology @xcite . dtms enable summarization, browsing, and searching of large document collections by capturing changes in topics over time. however, evaluating dtms can be challenging due to their unsupervised nature, although it is crucial for effectively detecting trends in time-indexed documents. with the advent of vae-based and llm-based topic models, there is an increasing need for eval-* equal contribution. uation procedures to compare these models @xcite . while traditional evaluation measures @xcite can assess the quality and diversity of topics, they fail to capture the smoothness of topic changes over time. this limitation becomes problematic when a dtm has high topic quality but lacks temporal smoothness. in such cases, existing evaluation measures may incorrectly assign a high score to the model, even when there are rapid and abrupt transitions between topics. for example, if a topic quickly changes from "politics" to "sports", conventional evaluation measures may still rate the model positively. to accurately assess the quality of a dtm, it is crucial to consider the smoothness of topic changes over time, which can help identify gradual topic drifts or sudden shifts. unfortunately, existing evaluation measures lack the ability to effectively track topic changes over time. to bridge this gap, we propose temporal topic quality (ttq)-a novel evaluation measure specifically designed for dtms. ttq incorporates changes in topic quality into its evaluation, thereby capturing the temporal characteristics of topics in dtms. we provide empirical evidence for the effectiveness of the proposed measure by evaluating it on both synthetic and real topics. the results demonstrate a positive correlation between human ratings and the individual components of the ttq measure. to provide an overall evaluation of dtms, we propose the dynamic topic quality (dtq). the dtq measure aggregates the ttq measure with the static topic quality score. this aggregation is performed for both year-wise evaluations and temporal topic assessments, as illustrated in figure 1 . in our experiments, we compare the results obtained using the dtq measure with those obtained using previously employed measures for different topic models, including llm-based topic models. we show that the dtq measure effectively indi-cates the smoothness of topics in trained dtms compared to the measures used in the past. we expect that the introduction of the new measure will contribute to improved comparisons between dtms in future research efforts. our contributions can be summarized as follows: ## related work this section presents the previous work on dtms and their evaluation approaches. we further discuss how the human evaluation for new measures was conducted in this domain. dynamic topic models are developed to model topics over time. dynamic latent dirichlet allocation (d-lda) @xcite extends the original lda method @xcite to account for temporal characteristics of sequential text data. the dynamic embedded topic model (d-etm) combines d-lda and word embeddings with a recurrent neural network (rnn) @xcite . sia et al. @xcite propose an llm-based topic model using pre-trained word embeddings and clustering them. while this is not a dynamic model that models topics over time, it is straight-forward to extend to a dynamic model by using an online clustering method. a continuous-time version of dtms was introduced by wang et al. @xcite . this model is not applicable to our datasets which have discrete timestamps. later work focused on the scalability of dtms due to their computationally intensive training @xcite . many other dtms have been proposed for different purposes, and not all are directly based on lda (grootendorst, 2022; @xcite @xcite @xcite @xcite @xcite . it can be expected that with the recent advent of neural topic models @xcite @xcite and llm-based topic models, more dtm variants will be published in the future. in this work, we compare d-lda and d-etm as the major proponents of the statistical and neural dtms. additionally, we evaluate an llm-based topic model @xcite . evaluation measures for topic models previous work has focused on measures for static topic models such as the topic coherence as measured by the normalized pointwise mutual information (npmi) or c v score @xcite @xcite . using a language model (llm) for evaluation @xcite is ineffective if the corpus is domain-specific and the llm has not been trained on that particular corpus @xcite . with the rise of neural topic models, local minima during training have become an issue, which may lead to component collapse @xcite . to capture such and other problems, different topic diversity, redundancy, or overlap measures have been introduced @xcite @xcite . dieng et al. combine topic diversity with coherence scores, resulting in topic quality scores @xcite , which is the basis of the definition of our temporal topic quality measure. human evaluation for topic models the concepts of coherence and interpretability are "simultaneously important and slippery" @xcite . a topic is coherent when a set of terms, viewed together, enables human recognition of an identifiable category @xcite . @xcite define an interpretable topic as one that can be easily labeled and has a high level of agreement on its labels. in the case of dtms, the topic coherence, interpretability, and smoothness across the temporal dimension are vital for its purpose and are the focus of our study. there are two main ways to carry out human evaluation of topic models: topic intrusion and topic rating. both were developed specifically to account for the topic coherence in static topic models. in the topic rating task, humans are presented with a topic and are asked to rate it on a scale. previously, topic 1 worker union labor contract strike worker union million jobs cuts budget billion workers government plan music dance opera ballet orchestra music theater opera jazz concert film theater movie production director topic 2 television news newspaper radio television news cable radio telephone news cable network telephone topic k 1987 1990 1995 money health budget state care television advertising media network computer internet technology card software 1997 𝑇𝑇𝑄 1 𝑇𝑄 1 𝑇𝑄 2 𝑇𝑄 3 𝑇𝑄 𝑇 𝑇𝑇𝑄 2 𝑇𝑇𝑄 𝐾 𝐷𝑇𝑄 = 𝑇 𝑇𝑄 𝑡 + 𝐾 𝑇𝑇𝑄 𝑘 authors have used ratings on a three-point ordinal scale @xcite @xcite @xcite . the rating task is not directly transferable to dtms since we need to also rate how the topic changes over time. in the topic intrusion task, topics are chosen randomly, and one word in the topic is replaced with a word from another topic. the intruder word @xcite is identified by human evaluators. here, we extend and tailor both tasks to the temporal evaluation of dtms. ## background on topic evaluation measures this section reviews the most common evaluation measures for topic models, which form the basis for our proposed measures: topic coherence, diversity, and quality. ## proposed measures none of the existing measures is suitable for evaluating temporal topic changes. our proposed measure fills this gap. first, we present the temporal topic coherence and smoothness measures. we then show how the two are combined to form temporal topic quality, which measures the quality of topic transitions over time. this measure is then used in an aggregated measure, the dynamic topic quality, which evaluates both crucial aspects in dtms: the quality of the topic model in each year and the quality of topic transitions over time. ## using human evaluation to examine dtms since topic modeling is unsupervised and no ground truth for topic quality is available, human evaluation is needed to validate our measures. in this section, we propose two tasks for the human evaluation of dtms. the proposed tasks are adapted from the widespread word intrusion @xcite and topic rating @xcite tasks for static topic models. ## experiments in this section, we establish the efficacy of the proposed measures using synthetic data and human evaluation. first, we compare static and temporal measures on synthetic topics in section 6.2. second, we investigate the sensitivity of the measures to noise in section 6.3. then, we conduct human evaluations of dynamic topics and compute the correlations of human ratings with the temporal measures in section 6.4. as a window size parameter for our proposed measures tts and ttc, we choose @xmath0. choosing higher window sizes would make the measure more sensitive to detecting slower transitions. however, sudden changes are of greater interest to us as they affect the interpretability of a topic over time more. 1 note, that in contrast to @xcite we do not use human evaluation here, but study correlation between intrusion level and coherence directly. ## conclusion this paper fills a gap in evaluating temporal characteristics of dtms such as llm-based dynamic models. we complement the existing year-wise measures by proposing novel temporal measures. our proposed temporal measures capture different aspects of temporal topic changes in dtms. we show that our measures are able to better capture temporal characteristics of topic changes than their year-wise counterparts and have positive correlation with human evaluations. we show the efficacy of our measure by evaluating different dynamic topic models and demonstrating their different temporal characteristics. our proposed evaluation measures will improve future comparisons between dtms, including llm-based topic evaluations. in the future, we want to extend this method also to the evaluation of sequential decision-making @xcite , other structured or online topic models @xcite as well as the explicit discovery of temporal anomalies @xcite .