Topic Modeling: Contextual Token Embeddings Are All You Need (2024)

Abstract: The goal of topic modeling is to find meaningful topics that capture the information present in a collection of documents. The main challenges of topic modeling are finding the optimal number of topics, labeling the topics, segmenting documents by topic, and evaluating topic model performance. Current neural approaches have tackled some of these problems but none have been able to solve all of them. We introduce a novel topic modeling approach, Contextual-Top2Vec, which uses document contextual token embeddings, it creates hierarchical topics, finds topic spans within documents and labels topics with phrases rather than just words. We propose the use of BERTScore to evaluate topic coherence and to evaluate how informative topics are of the underlying documents. Our model outperforms the current state-of-the-art models on a comprehensive set of topic model evaluation metrics.

## introduction the most widely used topic modeling approach is latent dirichlet allocation (lda) @xcite , which is a probabilistic generative model that models documents as a mixture of topics and each topic as a mixture of words. selecting the optimal number of topics is one of the primary challenges with lda and many other topic modeling approaches. additionally, lda uses bag-of-words (bow) representations of documents that ignore word semantics and syntax. lda models the underlying word distribution of documents, which necessarily makes uninformative words the most probable in topics, leading to poor topic interpretability. the authors of the lda paper make it clear: "we refer to the latent multinomial variables in the lda model as topics, so as to exploit text-oriented intuitions, but we make no epistemological claims regarding these latent variables beyond their utility in representing probability distributions on sets of words." @xcite . neural topic models use deep learning techniques to capture the syntax and semantics of text by leveraging word and document embeddings instead of relying on statistical inference on bow representations. the embedded topic model (etm) @xcite combined the power of lda with word embeddings. this approach overcomes the limitations of bow representation of documents and allows for richer semantic representation of words in a document. however, etm still ignores the syntax of words and requires the number of topics to be determined. the top2vec model @xcite ) introduced a novel method that leverages joint document and word embeddings to find topic vectors. it automatically finds the number of topics, it does not require stop-word removal and it also produces hierarchical topics. this approach has been shown to produce more informative and interpretable topics @xcite , @xcite , @xcite . top2vec captures word syntax with document embeddings and word semantics with word embeddings. its main limitation is that each document is assigned only one topic. the bertopic model @xcite uses bert @xcite document embeddings to find clusters of documents that are labeled with class-based tf-idf. the main limitations of this model are that it uses bow representation of document clusters which ignore word semantics, it does not assign topics to all documents, and it only assigns one topic per document. contextual embedding models (ctm) @xcite , @xcite extend neural product-of-experts lda (prodlda) @xcite by adding contextualized document embeddings. this model uses vector representation of words and documents so it therefore captures word semantics and syntax. its main limitation is that it requires the number of topics to be determined. the main advantage common to all the models discussed is that they improve topic coherence compared to lda. the main disadvantage is that they do not segment topics within a document; instead, they only give distributions of topics present in a document, with the exception of top2vec and bertopic which only assign one topic per document. another common disadvantage is that a topic distribution does not give information about the relevance of documents to a topic, it only gives the proportion of the document that is relevant. most topic evaluation metrics measure topic coherence, but little attention is given to how well the topics represent the underlying documents, which can lead to misleading results @xcite . accessing how well a topic represents the underlying documents is essential to evaluate the performance of a topic model @xcite . lastly, coherence measures designed for older models may not accurately represent the performance of novel neural methods @xcite , @xcite . ## conclusion we propose a novel topic modeling approach that leverages contextual token embeddings to create multi-vector document representations that capture topical information from each segment of a document. our method finds topic vectors and labels each topic using coherent and easily interpretable phrases. c-top2vec supports hierarchical topic reduction, enabling it to handle topics at various levels of granularity. our model segments documents into topic spans, enabling detailed and granular analysis of topics. our model produces a document topic distribution and a document topic relevance score which allows for ranking of document segments according to their relevance to a specific topic. we propose the use of bertscore for evaluating both topic coherence and how informative topics are of their underlying documents overcoming previous gaps in evaluation. our findings, based on a comprehensive set of topic modeling evaluation metrics, demonstrate that our model outperforms current state-of-the-art models.