H i COT : Improving Neural Topic Models via Optimal Transport and Contrastive Learning (2025)

Abstract: Recent advances in neural topic models (NTMs) have improved topic quality but still face challenges: weak document-topic alignment, high inference costs due to large pretrained language models (PLMs), and limited modeling of hierarchical topic structures. To address these issues, we introduce HiCOT (Hierarchical Clustering and Contrastive Learning with Optimal Transport for Neural Topic Modeling), a novel framework that enhances topic coherence and efficiency. HiCOT integrates Optimal Transport to refine document-topic relationships using compact PLM-based embeddings, captures semantic structure of the documents. Additionally, it employs hierarchical clustering combine with contrastive learning to disentangle topic-word and topic-topic relationships, ensuring clearer structure and better coherence. Experimental results on multiple benchmark datasets demonstrate HiCOT’s superior effectiveness over existing NTMs in topic coherence, topic performance, representation quality, and computational efficiency.

## introduction topic modeling has been recognized as a fundamental task in natural language processing (nlp), aims to uncover latent topic structures within a corpus, while simultaneously providing document topic distributions @xcite @xcite . in recent years, neural topic models (ntms) @xcite @xcite have emerged as a promising alternative, leveraging deep neural networks to enhance flexibility, improve the quality of discovered topics, help to overcome limitations about inefficient parameter inference of traditional topic models. despite these advancements, ntms still face several challenges. recent models exhibit limitations in effectively capturing document-topic relationships, lacking information and showing weak alignment between document representations and topic proportions. while some approaches leverage plms to enhance topic modeling @xcite , the process of extracting embeddings from large plms significantly increases inference costs, making them less practical for situations requiring low inference times. it is entirely feasible to replace plm-based embeddings with alternative methods that still achieve good performance, while significantly reducing computational overhead. to address the lack of document-topic relationships, we propose an apporach that effectively utilizes compact plms with a simple neural network. our method incorporates with optimal transport to refine relationships between documents and topics, enhancing the model's ability to learn more structured and semantic topic distributions through pretrained document embeddings. another limitation of ntms is their tendency to overlook hierarchical topic structure, limiting their ability to model semantic dependencies at different levels of abstraction. to tackle this issue, we introduce a hierarchical clustering framework combined with contrastive learning, explicitly disentangling relationships between topics and words, as well as between topics themselves. our approach regularizes semantic relations among topic and word embeddings, ensuring clear separation between topic clusters while maintaining meaningful relationships within each cluster. this leads to enhanced topic interpretability, coherence, and representation quality. in this work, we propose hicot (hierarchical clustering and contrastive learning with optimal transport for neural topic modeling), a novel framework that effectively addresses these challenges. we summarize the contributions of our study as follows: ## background consider a collection of bag-of-words (bow) representations, denoted as @xmath0, corresponding to d documents and defined over a vocabulary of v distinct words. topic modeling aims to uncover k latent topics within x. each topic k is associated with a topic-word distribution β @xmath1, forming a topic-word distribution matrix of k desired topics β = (β 1 , . . . , β k ) ∈ r v ×k . let l be the word embedding dimension, we define the word embedding matrix as @xmath2, . . . , w v ) ∈ r v ×l and the topic embedding matrix as @xmath3, . . . , t k ) ∈ r k×l . recent advances in neural topic modeling @xcite @xcite have shifted away from factorizing β as the product of word embeddings w and topic embeddings t . instead, β is expressed as: given a document x i , its bag-of-words (bow) representation is passed through an inference network to compute the parameters of a gaussian posterior distribution, where mean and diagonal covariance matrix are given by µ = h µ (x d , γ) and σ = diag(h σ (x d , γ)), respectively, with γ denoting the parameters of inference network. using the reparameterization trick (kingma and welling, 2013), the latent variable z is subsequently sampled from the posterior distribution q(z|x i ) = n (z|µ, σ). finally, the bow representation is reconstructed through β and θ, where the generative process follows a multinomial distribution xbow ∼ multi(softmax(βθ)). the loss function for topic modeling is composed of two components: a reconstruction term and a regularization term, as detailed below: unlike vae-based models, fastopic @xcite adopts a different inference mechanism. fastopic utilizes optimal transport to directly compute document-topic and topic-word distributions. specifically, it defines cost matrices c (1) ∈ r d×k and c (2) ∈ r k×v based on euclidean distances between document embeddings, topic representations, and word embeddings. the resulting optimal transport plans ψ * and ϕ * yield documenttopic and topic-word distributions θ = dψ * and β = kϕ * , with the overall loss: then, given a new document x ′ , fastopic performs inference by mapping it to an embedding d ′ and estimates its topic distribution θ ′ as follows: ## proposed method in this section, we present a detailed analysis of our proposed, which leverages compact plms and optimal transport to improve document-topic alignment. furthermore, we introduce a novel framework that incorporates hierarchical clustering and contrastive learning to reinforce topic-word associations and topic-level relationships. ## conclusion in conclusion, we introduce hicot, a novel neural topic modeling framework that integrates hierarchical clustering, contrastive learning, and optimal transport. by leveraging compact plms and optimal transport, hicot enhances document-topic alignment and captures semantic relationships between documents. additionally, contrastive learning strengthens both topic-word associations and inter-topic interactions, resulting in more coherent topic representations. comprehensive experiments on benchmark datasets demonstrate that hicot achieves superior performance in generating highquality topics and document-topic distributions, providing a robust and scalable solution for neural topic modeling.