Dynamic Structured Neural Topic Model with Self-Attention Mechanism (2023)

Abstract: This study presents a dynamic structured neural topic model, which can handle the time-series development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a self-attention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.

## introduction topic models are dominant tools for discovering the underlying semantic structure in a collection of documents. as a part of such topic models that can capture the chronological transition of topics have been intensively studied in recent years. the dynamic topic model (dtm; blei and lafferty, 2006) is a pioneering work that captures the time-series evolution of topics. it successfully visualizes the changes in the topic proportion and the word distributions of each topic over time. recently, neural networks have empowered topic models to handle a significant collection of documents. the dynamic embedded topic model (d-etm; @xcite introduces word embeddings and amortized variational inference into dtm, which significantly improves topic quality while reducing computational time. d-etm is widely applied to large-scale time-series documents, such as scientific papers and social media @xcite . however, dtm and d-etm assume that topics evolve independently without interaction. this assumption is inappropriate, particularly for mod-topic 1 ("#@xmath0) t-1 topic 1 (") topic 2 (") topic 1 ("&@xmath1) (a) t t+1 ## related work extending the dynamic topic model (blei and lafferty, 2006), several variants have been proposed, such as the dependent dirichlet processes mixture model @xcite , infinite dynamic topic model @xcite , and d-etm @xcite . however, these studies treat timeseries changes in topics independently and cannot capture dependencies among topics. relating to structured topic models, several models have been proposed such as tree-structured topic model @xcite @xcite and pachinko allocation models @xcite . in addition, several studies have modeled the structure among documents by jointly modeling citation network and text @xcite @xcite . however, these studies were not intended to track the time series transitions of topics. the dynamic and static topic model (dstm; @xcite extends the pachinko allocation model to capture the dynamic structure over time and the static structure among topics at each time step. dstm has several drawbacks against our model. it cannot capture topic dependencies across multiple time steps, and thus, incorporating citation information is challenging. in addition, it cannot predict emerging topics as topic transitions are parameterized for each time step. moreover, collapsed gibbs sampling is used to infer posteriors, which is not scalable for large datasets. the dynamic topic model on networked documents (netdtm; zhang and lauw, 2022) models time-series documents and citation networks simultaneously. however, netdtm does not capture the relations between topics nor predict emerging topics, which significantly differ from ours. contrary to the aforementioned studies, our work introduces amortized variational inference using the self-attention mechanism. this inference technique enables us to capture the topic branching and merging process across multiple time steps with significant scalability. furthermore, our model can predict emerging topics from past topics. ## background we first review the embedded topic model (etm; @xcite , and then explain d-etm, which combines etm with dtm before introducing our dsntm. ## dynamic structured neural topic model (dsntm) this section describes the proposed dsntm. the generative process of documents is the same as that of d-etm. ## conclusion in this study, we proposed a novel dynamicstructured neural topic model, dsntm, which captures dependencies among topics using the selfattention mechanism. we also introduced a citation regularizer, which induces the attention weights to correspond to citation relations. experimental results demonstrated that dnstm outperforms previous dynamic topic models regarding perplexity and coherence while maintaining sufficient diversity across topics. in addition, dsntm can identify the process of topic merging and branching while showing the potential to predict emerging topics. we expect that dsntm will make it easier for non-specialists to keep track of the evolution of topics in a given research area without retracing the citations of copious articles and assist their search for a novel topic.