Dynamic Topic Modeling by Clustering Embeddings from Pretrained Language Models: A Research Proposal (2022)

Abstract: A new trend in topic modeling research is to do Neural Topic Modeling by Clustering document Embeddings (NTM-CE) created with a pretrained language model. Studies have evaluated static NTM-CE models and found them performing comparably to, or even better than other topic models. An important extension of static topic modeling is making the models dynamic, allowing the study of topic evolution over time, as well as detecting emerging and disappearing topics. In this research proposal, we present two research questions to understand dynamic topic modeling with NTM-CE theoretically and practically. To answer these, we propose four phases with the aim of establishing evaluation methods for dynamic topic modeling, finding NTM-CE-specific properties, and creating a framework for dynamic NTM-CE. For evaluation, we propose to use both quantitative measurements of coherence and human evaluation supported by our recently developed tool.

## introduction the ever-accelerating pace at which online documents, and specifically text documents, are published creates a need for methods able to analyze text documents in large quantities, something topic models were created to do. in this paper, the term document refers to a sequence of words in natural language, such as a tweet or a news article. the topic model analyzes a collection of documents to discover the major topics appearing in it. then, each document is related to the discovered topics. successful topic modeling applications cover a wide range of fields, such as studying historical documents @xcite , discovering gender bias in datasets @xcite , and catching new trends on social networks @xcite . topic models that do not consider the temporal dimension of a document collection are called static topic models. however, an important aspect missed by such models is the evolution of topics or different temporal contexts a document can be situated in. for example, a topic centered around diabetes will have a different discussion before and after the discovery of insulin, and the topics surrounding ukraine have shifted rather dramatically during 2022. models that consider the temporal dimension are called dynamic topic models (dtms). this paper proposes an extensive study on how to efficiently create dtms based on neural topic models. neural topic models (ntms) are topic models that are created with the help of neural networks @xcite . they became competitive with the advances in language modeling in the previous decade. an ntm with an incorporated pretrained language model (plm) is called ntm-plm. a pretrained language model, such as bert @xcite , is capable of embedding words or documents into a vector representation that reflects aspects of the meaning of the text and thus its relation to other texts. we define a document embedding to be a mapping of the document collection to a vector space in such a way that the vector representing a given document captures some sort of information about the document. the information should largely be obtained using the plm, but can also blend with other information if there is extra input such as timestamps. since each two embedding vectors are at a certain well-defined distance from each other in the vector space, the conceptually most straightforward approach to do topic modeling is to apply a distance-based clustering algorithm to the embeddings. in this paper, we call this procedure neural topic modeling by clustering embeddings (from pretrained language models), ntm-ce. the distinction between our definition of ntm-ce and other models within the ntm-plm sphere is how the topics are formed. for a model to be an ntm-ce, the topics must be formed by applying a distance-based clustering algorithm to the embeddings that were created by the plm. we define the core pipeline of ntm-ce as @xmath0. the most common ntm-ce methods that we are aware of use a dimension reduction technique as a vector transformation before clustering. however, we also consider other transformations that could be applied to enhance the vector space to discover meaningful topics. belonging to our definition of ntm-ce models are cetopic @xcite and bertopic (grootendorst, 2022), but not models such as embedding topic model (etm) @xcite or ze-roshottm @xcite since the latter do not directly cluster the embeddings. ntm-ce models have shown promising performance @xcite @xcite when compared to classic generative methods such as latent dirichlet allocation (lda, @xcite ). moreover, ntm-ce are gaining traction due to their conceptual simplicity and modularity. early studies compare the topic coherence of ntm-ce with other established models to legitimize its use. however, an understudied part of these new topic models is their ability to do dynamic topic modeling. having a conceptually simple dtm which improves with advancements in language modeling is attractive for many research communities and industries whose data tends to end up consisting of large unstructured sets of documents collected over time. dynamic topic modeling may, for instance, enable a company to discover and react to trends before they become mainstream. an important aspect of research on dtms is how to evaluate such models. for the evaluation of static models, there has been substantial research on how different quantitative measurements relate to topic quality and human judgment @xcite . in contrast, little research has been devoted to the fair comparison of dtms. therefore, we suggest to develop a framework to make such comparisons. in this proposal, we present some potential ways in which such a comparative evaluation of dtms could be done. while ntm-ce is a promising technique, there is little research on adding a temporal dimension to ntm-ce and how to fairly compare dtms. therefore, we propose the following research questions in an attempt to thoroughly investigate the prospect of using ntm-ce for dynamic topic modeling. rq1: what requirements exist for a dynamic topic modeling system and how can ntm-ce properties respond to those requirements? rq2: how viable is dynamic topic modeling with ntm-ce in practice? with rq1 we aim to lay a theoretical foundation that will lead to knowledge that remains valid beyond the current state of the art in language modeling. the goal is to create a general framework and to thoroughly address the strengths and weaknesses of ntm-ce from a dynamic perspective. rq2 puts theoretical knowledge into practice and will reveal insights and limitations of dynamic ntm-ce. here, we strive to create functional models that can solve problems of practical relevance for academia and industry. ## literature review topic modeling is a field within text mining whose objective is to find topics that best describe a collection of documents and then assign the documents to these topics. models from the stochastic school use latent dirichlet allocation (lda) @xcite and its variants to fulfill this objective using probabilistic distributions to discover topics. the field of dynamic topic modeling (dtm foot_0 ) is the study of topics evolving over time. discrete dtm such as discrete lda (d-lda) @xcite divides a topic into batches of discrete time steps where the next time step in a topic evolves from the previous. continuous dtm (c-dtm) as introduced by @xcite borrowed the concept of brownian motion from physics, which makes it possible to view the evolution of topics in continuous time. another model that works continuously is topics over time (tot) by @xcite which associates each topic with a beta distribution representing the temporal dimension. in our project, we wish to study how to create a continuous-time topic model with ntm-ce as its basis. in addition, we want to use the dynamic models mentioned above for benchmarking future models. with the introduction of word2vec @xcite , topic modeling saw a new era of models that make use of embeddings. the surveys by @xcite and @xcite describe how topic modeling meets neural networks in the modern era. they also describe many models that incorporate plms. dieng et al. (2020) et al. (2021b) and bianchi et al. (2021a) describe other examples of models that incorporate embeddings created by plm (e.g. s-bert, @xcite ) to a traditional topic modeling structure, but without using a distance-based clustering algorithm to determine topics. models that directly cluster vectors in the vector space created by a plm are what we call ntm-ce. the popularity of this approach increased after the release of bert @xcite , and most models since then have used bert or a bert variant. @xcite used bert, principle component analysis (pca) @xcite , and k-means clustering @xcite and found the ntm-ce pipeline to perform similarly to lda. @xcite compared different flavors of bert and gpt-2 @xcite in combination with pca and k-means, also concluding that the technique performs better than lda. using more recent libraries, @xcite proposed bertopic, which uses bert, umap @xcite @xcite together with a novel term-weighting procedure c-tf-idf to discover topics in news data. similarly, @xcite use bert, umap, and k-means together with term weighting and conclude that the model outperforms all previous models. while all of these models use plm embeddings, there is no consensus as to which embeddings to cluster. @xcite clusters vocabularylevel embeddings, @xcite clusters token-level embeddings and grootendorst (2022) clusters sentence-level embeddings. part of the current proposal is to investigate differences in properties between different embedding choices and to study whether one of them is preferable for ntm-ce. taking another perspective, @xcite jointly train the dimension reduction and clustering components to obtain a vector space with high clusterability in the sense of @xcite . their model topclus was highly successful and thus an interesting testament to what can be done when combining components. however, to limit the research plan described here, we will initially focus on approaches with separately trained components, saving jointly trained ones for later. ## proposed work in this section, we present the proposed work to seek answers to the research questions posed in section 1. the research is divided into four phases, expected to correspond to 1-2 papers each. the work is to be done over two years. before explaining the phases, an outline of a potential system that is supposed to accompany the theoretical work is given. ## evaluation evaluation of topic models is not trivial as the lack of an objective ground truth makes it hard to achieve a consensus on the number and nature of topics. automatic measurements for evaluating topic models are topic coherence and topic diversity. while there are many approaches, topic coherence is usually automatically measured using normalized point-wise mutual information (npmi, @xcite ) and is considered by some to mimic human judgment @xcite . topic diversity measures in different ways how diverse the top words in a topic are to each other @xcite @xcite . there are tools like octis @xcite that make it easy to compare static topic models. we plan to extend octis or use similar ideas to facilitate fair comparisons between dtms. as a first step, an octis extension may average topic coherence and topic diversity over the time steps as has been done by @xcite and @xcite . furthermore, part of the work in phase i will be to assess how to measure aspects more specific to dynamic modeling requirements. this could, e.g., result in the requirement to develop an initial benchmark dataset for topic change over time, or to find a way to quantitatively assess topic change without a ground truth. the importance of automatic measurements that correlate with human judgment has been known @xcite and npmi was adopted after showing such a correlation. however, a recent study by @xcite argues that automatic coherence measurements, including the prevailing topic model evaluation standard npmi, should not be considered equivalent to human judgment. therefore, we plan to complement automatic measurements with human evaluation when resource allocation is justified, for example, when a core pillar of the work needs to be validated. qualitatively, we assume that a human can look at a sample of topics produced by a topic model and decide if they think the topics are coherent and also which documents should not be considered to belong to the topic. we use this assumption to develop a tool for rapid human evaluation of topics, which we intend to make use of to validate automatic measurements. this tool is described further in section 5 evaluation; see figure 2 . the idea behind it is that the coherence of a topic can be more confidently assessed if an evaluator reads the actual titles and text of the articles rather than only a few describing keywords. by having the evaluator systematically go through all topics, we can get a score for how well the model performs from a human perspective. this requires expert evaluators who can contextualize a given number of articles and then also mark articles that do not belong to the topic. stellar is supposed to aid the evaluation process and make it faster. the core functionalities are a topic list, an article list from the chosen topic, a box for reading the article text body, and a 3d visualization of the document vector space. in the article list, the articles can be marked as not belonging to the topic. the tool will be extended to make it easier to analyze dynamic topic models. the extension could add functionality like selecting articles within different time periods, visualizing the varying size of topics over time, or visualizing changes in the topic description over time. the functions and statistics that are needed we expect to become clear over the course of working with this project. ## impact in the current information era, there are obvious benefits to having fast and trusted topic models which can process large corpora of documents. we have seen that there is a wide range of applications such as analyzing historical documents, social media, or news articles. research on ntm-ce is particularly interesting because of the modularity of the different components, especially to isolate the language model in the system. this allows for exchanging parts of the system when new and better components are developed, meaning that this type of topic modeling will continue to improve even after the popular language model at the time is superseded by a better one. developing a solid framework for dynamic topic modeling with this modularity will ensure that ntm-ce models are as flexible as lda-based models in their applications. ## summary this paper presented an outline to explore the dynamic topic modeling in detail with ntm-ce. we proposed two research questions in an attempt to cover both the theoretical framework and practical application of dynamic topic modeling with ntm-ce. we propose four phases in which the work will be done and where the main contributions will be: 1) a codebase for evaluating dynamic topic models, 2) a general framework for how to efficiently create dynamic topic models with ntm-ce, and 3) insights from practical application of the framework to various datasets. the research questions and phases were developed to the best of our ability with our current understanding. however, we see them as constantly evolving as we learn more. therefore, we highly welcome all types of input from the research community to make this project as relevant and impactful as it can be.