HTMOT : Hierarchical Topic Modelling over Time (2023)

Abstract: Topic models provide an efficient way of extracting insights from text and supporting decision-making. Recently, novel methods have been proposed to model topic hierarchy or temporality. Modeling temporality provides more precise topics by separating topics that are characterized by similar words but located over distinct time periods. Conversely, modeling hierarchy provides a more detailed view of the content of a corpus by providing topics and sub-topics. However, no models have been proposed to incorporate both hierarchy and temporality which could be beneficial for applications such as environment scanning. Therefore, we propose a novel method to perform Hierarchical Topic Modelling Over Time (HTMOT). We evaluate the performance of our approach on a corpus of news articles using the Word Intrusion task. Results demonstrate that our model produces topics that elegantly combine a hierarchical structure and a temporal aspect. Furthermore, our proposed Gibbs sampling implementation shows competitive performance compared to previous state-of-the-art methods.

## introduction in the field of natural language processing (nlp), numerous methods for extracting topics from a corpus have been proposed over the years @xcite . while the seminal latent dirichlet allocation (lda) algorithm @xcite paved the way for topic modeling, it lacks the ability to capture hierarchical or temporal information. in the past, hierarchical topic models have been proposed @xcite that enable the extraction of topics and sub-topics organized in a tree-like structure. these models dynamically determine the appropriate number of topics and sub-topics during training and have been found to be useful in ontology learning @xcite and research idea recommendation @xcite . in parallel, temporal topic models have been developed @xcite @xcite that allow for the extraction of topics that describe events or trends occurring in a corpus. these models have been applied to tasks such as tracking trends in scientific articles @xcite and events in social media @xcite . combining hierarchical and temporal information in models can capture broad and detailed aspects of a corpus, benefiting applications like environment scanning @xcite . hierarchical modeling yields detailed topics and subtopics for a comprehensive thematic understanding, while temporal modeling provides precise descriptions of events. this integration produces nuanced models for informed decision-making and deeper insights. however, integrating temporal and hierarchical information in topic models remains a challenge @xcite @xcite . many temporal models have their own structures to represent time, such as time trees or time slices, which complicates the integration with a hierarchical structure @xcite @xcite . the only temporal model that does not require its own structure is tot @xcite , but combining time and hierarchy is still difficult due to the beta distribution used to model time lacking a known conjugate prior, making it incompatible with stochastic variational inference (svi) used by previous hierarchical models @xcite . our proposed method, hierarchical topic modelling over time (htmot), jointly models topic hierarchy and temporality to leverage the strengths of both dimensions and to overcome the challenges associated with integrating them. as a secondary contribution, we propose a novel implementation of gibbs sampling based on a treebased data structure called the infinite dirichlet tree. this implementation is comparable to svi in terms of speed. our work provides a promising avenue for addressing the need for topic models that can incorporate both hierarchical and temporal information. @xcite we performed our experiments using a corpus of 62k news articles and evaluated our method using the word intrusion task @xcite . ## related work we now describe previous topic modelling methods most closely related to ours. for more comprehensive reviews see @xcite and @xcite . ## results and discussion we now present our results, starting with a statistical analysis of the training behavior of htmot. then, we will discuss the results of the word intrusion task, its drawbacks, and directions for future topic modeling evaluation methods. finally, we will examine the various extracted topics qualitatively. ## conclusion we have proposed a new model for topic modeling capable of modeling hierarchy and time jointly. through examples, we have demonstrated how combining hierarchy and temporality provides us with a more fine-grained understanding of a corpus through detailed sub-topics which can represent specific events. moreover, we developed a novel implementation of gibbs sampling for hierarchical topic models. this implementation provides a fast alternative to svi that makes gibbs sampling a viable solution for training such complex models. moreover, we have shown how extracting entities can help interpret and understand topics at a deeper level.