Topic Modeling With Topological Data Analysis (2022)

Abstract: Recent unsupervised topic modelling ap-proaches that use clustering techniques onword, token or document embeddings can ex-tract coherent topics. A common limitationof such approaches is that they reveal noth-ing about inter-topic relationships which areessential in many real-world application do-mains. We present an unsupervised topic mod-elling method which harnesses TopologicalData Analysis (TDA) to extract a topologicalskeleton of the manifold upon which contextu-alised word embeddings lie. We demonstratethat our approach, which performs on par witha recent baseline, is able to construct a networkof coherent topics together with meaningfulrelationships between them.

## introduction unsupervised topic modelling is a standard technique for making sense of document collections. while traditional approaches such as lda @xcite rely on probabilistic models, the field has recently moved towards clustering-based methods in which topic clusters are obtained via document, word or token embeddings @xcite @xcite . even though clustering can yield interpretable topics, it typically discards information about relationships between clusters, hence making it harder to interpret clusters in global contexts. in this work, we approach topic modelling as a task to find regions on a manifold of contextualised word embeddings which reflect a "topic". to this end, we apply mapper -an algorithm from the field of topological data analysis (tda). mapper creates a graph whose topology reflects the shape of the underlying data set and whose nodes represent subsets of data points. in the case of contextualised word embeddings, we construct a graph where each node is a cluster of tokens (i.e. a "topic"), and where connections between them reflect the topology of the embedding manifold. we use community detection techniques to demonstrate that semantically related topics are connected in the graph. our main contributions are the following: ## related work the seminal work on unsupervised topic modelling was @xcite who introduced latent dirichlet allocation (lda), a bayesian generative model of documents which assumes that the tokens in a document are drawn from a mixture model whose mixture components are interpreted as topics. of the many extensions to the classic lda archetype that have since been proposed, most relevant to our present work are methods to model associations and relationships between topics, and the use of neural representations in general and contextualised representations in particular. correlated topic models @xcite are lda extensions that attempt to learn the structure of topic associations within a document. the goal of hierarchical topic models @xcite @xcite @xcite @xcite is to learn a treestructured graph of topics by incorporating hierarchical non-parametric bayesian priors into traditional topic models. several studies have combined topic modelling with neural representations with a view to learn better topics or representations. for example, amortised variational inference with neural variational posteriors (kingma and welling, 2014) has been investigated as a means to scale up inference on probabilistic topic models and relax the conjugacy assumptions which are required for tractable inference in traditional topic models @xcite . various variants of such models have focused on neural extensions of correlated @xcite and hierarchical @xcite topic models although they all use neural representations in the generative model or variational posterior. some studies have also incorporated contextualised word embeddings into topic models while still using neural probabilistic generative models @xcite . the prior work most closely related to our proposed method is the joint application of topic modelling and contextualised word embeddings by @xcite @xcite who induce topics via vector clustering over word or document embeddings. our method differs from lda and its extensions in that we use tda rather than probabilistic generative models to induce topics. correlated topic models and their neural extensions learn a flat topic structure while adding scalar associations, whereas our method induces a topic graph. in contrast to hierarchical topics models and their neural extensions which induce tree-structured topic graphs, our method induces an unrestricted graph. unlike our method, previous work on inducing topics from contextualised word representations construct a flat topic structure rather than a graph. also related to our work is topoact @xcite which applies mapper to the analysis of bert word embeddings. our work differs from ibid. in that we focus specifically on topic modelling, and we follow a systematic hyperparameter selection process through stability analysis. ## proposed method the manifold hypothesis states that real-world highdimensional data lie on a low-dimensional manifold embedded in a high-dimensional space. topic modelling can be regarded as an endeavour to identify topologically meaningful regions of the word representation manifold which contain homogeneous topics or words. traditionally, it has been approached as a clustering problem in that the representation manifold is assumed to be a disconnected union of "topic" manifolds. however, such an assumption is clearly limiting and not grounded theoretically. one potential solution involves dimensionality reduction and direct manifold visualisation. unfortunately, most dimensionality reduction techniques capture only topology within local neighbourhoods, and cannot be relied upon for inference regarding the global topology of the manifold. our method of choice to address this problem is tda mapper introduced in @xcite (also referred to as topological data visualisation or topological clustering), a method that yields an approximation of a reeb graph of a manifold @xcite which captures the topology and shape of the manifold. reeb graphs are constructed from a manifold in order to learn topological invariants and global structure. even though they lose some of the original topological structure of the manifold, their low-dimensional invariants (e.g. connected components) remain the same. ## results ## conclusion we propose an unsupervised topic modelling method which leverages topological data analysis (tda) to extract a semantic topic graph from a large unstructured document collection. our experimental results demonstrate that our method is able to detect topics on par with a recent baseline while also exposing meaningful inter-topic relationships towards deeper topic interpretation. our experiments to date motivate future work involving tda to develop, for example, interactive visualisation tools for exploring rich relational topic graphs, and to study the interface between topological and linguistic properties of topics.