Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion (2024)

Abstract: Dynamic topic models track the evolution of topics in sequential documents, which have derived various applications like trend analysis. However, existing models suffer from repetitive topic and unassociated topic issues, failing to reveal the evolution and hindering further applications. To address these issues, we break the tradition of simply chaining topics in existing work and propose a novel neural Chain-Free Dynamic Topic Model. We introduce a new evolution-tracking contrastive learning method that builds the similarity relations among dynamic topics. This not only tracks topic evolution but also maintains topic diversity, mitigating the repetitive topic issue. To avoid unassociated topics, we further present an unassociated word exclusion method that consistently excludes unassociated words from discovered topics. Extensive experiments demonstrate our model significantly outperforms state-of-the-art baselines, tracking topic evolution with high-quality topics, showing better performance on downstream tasks, and remaining robust to the hyperparameter for evolution intensities.

## introduction dynamic topic models seek to discover the evolution of latent topics in sequential documents divided by time slice. for example, figure 1 illustrates topic#1 about covid-19 and topic#20 about ukraine evolve from 2020 to 2022 in the documents divided by year. this evolution reveals how topics emerge, grow, and decline due to trends and events. such evolution has been employed in various downstream applications, e.g., opinion mining, trend tracking, and sentiment analysis @xcite @xcite @xcite @xcite . existing dynamic topic models can be classified into two types: (i) probabilistic dynamic topic models @xcite , learning through variational inference @xcite or gibbs sampling @xcite , and (ii) neural dynamic topic models @xcite @xcite , learning via gradient back-propagation. however, these existing models mostly chain topics via markov chains to capture topic evolution, suffering from two vital issues: (i) repetitive topics: topics within a time slice are repetitive with similar semantics. to solve the above issues, we break the tradition of chaining topics via markov chains and propose a novel neural chain-free dynamic topic model (cfdtm). first, we propose evolution-tracking contrastive learning (etc) to address the repetitive topic issue. etc adaptively builds positive and negative relations among dynamic topics. building positive relations tracks the topic evolution with different intensities. more importantly, building negative relations encourages topics within each slice to be distinct, maintaining topic diversity and thus alleviating the repetitive topic issue. second, we propose a new unassociated word exclusion (uwe) method to solve the unassociated topic issue. uwe finds the top related words of topics at each time slice and identifies which of them do not belong to this slice as unassociated words. then uwe explicitly excludes these unassociated words from topics to refine topic semantics, such as excluding the unassociated word "covid" from topic#2 in 2017 in neural dynamic topic modeling due to the success of neural topic modeling @xcite @xcite @xcite , neural dynamic topic modeling has attracted more attention @xcite . dieng et al. (2019) first propose detm in the framework of vae (kingma and welling, 2014; rezende et al., 2014). later, zhang and lauw (2022) capture evolution from temporal document networks; cvejoski et al. (2023) focus on modeling topic activities over time; rahimi et al. (2023) cluster documents to find dynamic topics but cannot infer their topic distributions. miyamoto et al. (2023) model the dependencies among topics with an attention mechanism. they all chain topics via markov chains following probabilistic models. differently instead of markov chains, we propose the novel evolution-tracking contrastive learning and unassociated word exclusion methods to track topic evolution and address the repetitive topic and unassociated topic issues. contrastive learning the goal of contrastive learning is to learn the similarity relations among samples @xcite @xcite @xcite @xcite . it has become a prevalent self-supervised fashion in vision and textual fields @xcite @xcite . several studies apply contrastive learning in static topic models @xcite @xcite @xcite @xcite . rather than these static topic models, we focus on dynamic topic models, which motivates our new evolutiontracking contrastive learning. ## methodology we recall the problem setting of dynamic topic modeling and present our novel evolution-tracking contrastive learning and unassociated word exclusion. finally we introduce the chain-free dynamic topic model (cfdtm) on these two methods. ## experiment in this section, we conduct comprehensive experiments and show that our model achieves stateof-the-art performance and effectively solves the repetitive topic and unassociated topic issues. ## conclusion in this paper, we propose cfdtm, a new chainfree neural dynamic topic model. to break the tradition of simply chaining topics in previous work, cfdtm employs the novel evolution-tracking contrastive learning and unassociated word exclusion methods. experiments demonstrate that our cfdtm consistently outperforms baselines and effectively mitigates the repetitive topic and unassociated topic issues. cfdtm tracks topic evo-lution with higher coherence and diversity, and achieves better performance on downstream tasks. our model also shows robustness to the hyperparameter for evolution intensities.