Unified Neural Topic Model via Contrastive Learning and Term Weighting (2023)

Abstract: Two types of topic modeling predominate: generative methods that employ probabilistic latent models and clustering methods that identify semantically coherent groups. This paper newly presents UTopic (Unified neural Topic model via contrastive learning and term weighting) that combines the advantages of these two types. UTopic uses contrastive learning and term weighting to learn knowledge from a pretrained language model and discover influential terms from semantically coherent clusters. Experiments show that the generated topics have a high-quality topic-word distribution in terms of topic coherence, outperforming existing baselines across multiple topic coherence measures. We demonstrate how our model can be used as an add-on to existing topic models and improve their performance.

## introduction one of the most common tasks in natural language processing (nlp) is to find cohesive topics in a text corpus. topic modeling is widely used in various applications, including trend extraction from real-time streams such as social media @xcite and identification of notable events upon risk @xcite . two representative lines of work exist: generative methods and clustering-based methods. generative methods follow the assumption of probabilistic latent semantic analysis (hereafter p-lsa) such that every word token in a document is sampled from a mixture of latent topics @xcite . such methods estimate both the latent topic distribution per document and word distribution per topic from bag-of-words (bow) representation @xcite @xcite . latent dirichlet allocation (lda), for example, uses a bayesian model to estimate the latent topic distribution (blei et al., * corresponding authors 2003). variational autoencoder (vae) retrieves latent topics from the topic distribution while also recovering the original input @xcite . recent techniques have employed pretrained language models on top of vae-based models to improve topic quality @xcite . generative probabilistic models work on the assumption that each topic is a mix of words from a larger set. however, they share a common weakness: the bow representation only contains word-level co-occurrences and fails to capture each word token's importance in the document's context information. consequently, the quality of the estimated topic is reliant on the choice of the word set. clustering methods, on the other hand, regard topics as semantic clusters discovered over the document or word embedding space @xcite @xcite . they utilize the knowledge of the pretrained language model (e.g., bert @xcite ) to generate highlevel summaries of given documents (or words). then, document (or word) clusters are identified according to the embedding distance via clustering methods, such as dbscan or k-means. they do not require word selection and effectively leverage context information for discovering topics. nevertheless, their core is clustering; hence, they cannot assign a mixture of topics to each document. this research takes a step further by combining the benefits of the previous methods by including novel considerations: contrastive learning and term weighting. we start with a generative method and add a term weighting scheme that mimics the clustering method. we use a contrastive learning framework to help the language model instill pretrained knowledge and enable the model to focus on influential words. our model, utopic (unified neural topic model via contrastive learning and term weighting), has three stages: (1) identifying semantic clusters, (2) calculating term weights, and (3) estimating the latent topic distribution. in stage 1, input documents with similar meanings are grouped into a single cluster. this procedure uses a pretrained language model, which extracts meaningful embeddings from the input document. stage 2 computes the term weights for each word to represent the importance of each term, where high weights are given to frequent terms in one semantically coherent cluster (but not in other clusters). the model selects words with top-k term weights as the final word set for bow representations. in stage 3, the model estimates the latent topic distribution by reconstructing bow representations from inputs. term weights can represent each term's importance, allowing the model to learn a coherent topic. contrastive learning manages the entire process of instilling pre-trained knowledge from the language model and generating a more distinct topic distribution with a predefined prior. experiments show that utopic outperforms conventional methods in a wide range of scenarios, including human-annotated datasets ( codes and implementation details of the model are available at a github repository. 1 1 https://github.com/mingi-sid/utopic 2 related works topic modeling algorithms can be divided into two major streams: generative and clustering-based. ## methods problem definition: let x be an input document and x be a collection of documents in the corpus (i.e., @xmath0. a primary goal of our topic model is to estimate the topic distribution p from x . given a finite number of topics k, we extract top-k relevant words (hereafter referred to as topic words) w i for each topic i to interpret the underlying theme and evaluate the topic quality. the topic word comes from the word set w dict , which has a size n (i.e., w i ⊂ w dict for all i). we propose to compute the weight importance of each term on semantically coherent clusters and then estimate the latent topic distribution by reconstructing the bow representations based on term weights. our idea is depicted in figure 1 , where a clustering model groups documents with similar contexts in the corpus (section 3.1). term weights are computed by treating each document cluster as a single unified document. if certain terms are significant in one group but not in others, we consider them influential. top-k influential subject terms are selected as the word set w dict for constructing bow representation (section 3.2). the model estimates topic distribution using the discovered word set and its term weight to recover the bow representation of documents (section 3.3). contrastive learning (hereafter cl) improves representation quality at every phase, and the dirichlet prior controls the entropy of the topic distribution. these steps are outlined in the following sections. ## conclusion this work proposed a new way to leverage the benefits of combining generative and clustering methods of topic modeling into a single framework. diverse topic coherence measures have been used to assess the quality of topics. compared to baselines that excel in only one or two coherence measures, utopic showed consistent improvement across multiple coherence measures and discovered topics that align well with human annotations. our method has a wide range of applicability because it can be added as a module to other existing approaches.