Supervised Neural Topic Modeling with Label Alignment (2025)

Abstract: Neural topic modeling is a scalable automated technique for text data mining. In various downstream tasks of topic modeling, it is preferred that the discovered topics well align with labels. However, due to the lack of guidance from labels, unsupervised neural topic models are less powerful in this situation. Existing supervised neural topic models often adopt a label-free prior to generate the latent document-topic distributions and use them to predict the labels and thus achieve label-topic alignment indirectly. Such a mechanism faces the following issues: 1) The label-free prior leads to topics blending the latent patterns of multiple labels; and 2) One is unable to intuitively identify the explicit relationships between labels and the discovered topics. To tackle these problems, we develop a novel supervised neural topic model which utilizes a chain-structured graphical model with a label-conditioned prior. Soft indicators are introduced to explicitly construct the label-topic relationships. To obtain well-organized label-topic relationships, we formalize an entropy-regularized optimal transport problem on the embedding space and model them as the transport plan. Moreover, our proposed method can be flexibly integrated with most existing unsupervised neural topic models. Experimental results on multiple datasets demonstrate that our model can greatly enhance the alignment between labels and topics while maintaining good topic quality.

## introduction topic modeling @xcite aims to uncover the latent semantic structure within a corpus, which has been applied in various downstream tasks, such as social event analysis @xcite and opinion mining @xcite . a topic is typically described as a probability distribution over words. when conducting topic modeling, we often focus on the extent to which the discovered topics are meaningful and easy to understand, i.e., the interpretability of topics. to assess it, the topic coherence and many other evaluation have been developed @xcite . however, a high coherence score of topics is often insufficient to indicate that they are ''good'' topics @xcite . considering that the human-defined labels describe some inherent patterns within the documents. we expect to identify the relationships between labels and the discovered topics, namely, achieve label-topic alignment. this will facilitate better understanding of the discovered topics, especially in the area of content analysis @xcite . for example, the movie reviews in the imdb dataset @xcite are categorized by their sentiment orientations (positive or negative), and we are interested in what a ''positive/negative'' topic is like. unfortunately, due to the lack of guidance from manually defined labels, it is challenging for topics discovered by unsupervised topic models to be consistent with them. in response to this, several supervised topic models have been developed @xcite . the fundamental idea behind most of them is to predict labels through document-topic distributions. however, they are faced with the following challenges: 1) a label-free prior p(z), which treats each topic evenly, is often adopted to generate the latent topics z i for each word (see figure 1a ). as the generated topics are not specific to the label y, this will blend the semantics of multiple categories. 2) under the label-free prior, they are not able to construct the label-topic relationships explicitly and thus fail to provide an intuitive view of the relationships between labels and topics. while there are statistical topic models based on gibbs sampling that explicitly construct the label-topic relationships @xcite , they suffer from inflexible model-specific derivations and slow inference speed. they are not neural topic models (ntms) and do not leverage the advantages of deep generative models like variational autoencoder (vae) @xcite . given the challenges mentioned above, it is desirable that a label-conditioned prior is utilized. under such a prior p(z|y), label-specific topics will be generated (see figure 1b ). motivated by this, we propose a novel supervised ntm, label aligned neural topic model (lantm). 1 first, within the framework of vae, we introduce a novel chain-structured graphical model that incorporates a label-conditioned prior. second, we propose to conduct inference under soft label-topic indicators. in order to obtain well-organized label-topic relationships, we then formalize an entropy-regularized optimal transport (ot) problem on the embedding space and model the label-topic relationships as the transport plan. besides, we propose to reconstruct the label-specific pseudo-documents to allow the label-topic distributions to capture the semantics of labels. the main contributions of this paper are as follows: ## lantm: label aligned neural topic model in this section, we propose lantm, a novel supervised ntm for label-aligned topic discovery. we start by discussing the generative process. we then introduce soft indicators to model the variational distribution and the label-conditioned prior. in order to enforce well-organized label-topic relationships, we formalize an entropy-regularized optimal transport (ot) problem and model the label-topic relationships as the transport plan. ultimately, we describe the parameter inference for lantm, and discuss how to conduct inference in the absence of labels. ## conclusion in this paper, we propose a novel supervised ntm called lantm, which overcomes the limitation in existing supervised ntms brought by a label-free prior. our lantm utilizes a chain-structured graphical model with a label-conditioned prior. the inference is conducted under soft indicators that describe the relationships between labels and topics. to obtain well-organized label-topic relationships, we formalize an entropy-regularized ot problem and model the label-topic relationships as the transport plan. we propose to reconstruct label-specific pseudo-documents to enable the label-topic distributions to capture the label semantics. our lantm can be seamlessly integrated with most existing unsupervised ntms. experimental results show that our model can effectively enhance the label-topic alignment while maintaining topic quality. in future work, we will expand our method to model more fine-grained dependencies such as label's dependencies. a derivation of the elbo log p(x, y) = log e q(z|x,y) p(x, y, z) q(z|x, y) (15) ≥ e q(z|x,y) [log p(x, y, z)] -e q(z|x,y) [log q(z|x, y)] (16) = e q(z|x,y) [log p(x|z)] + e q(z|x,y) [log p(z|y)] + e q(z|x,y) [log p(y)] -e q(z|x,y) [log q(z|x, y)] (17) = e q(z|x,y) [log p(x|z)] -d kl [q(z|x, y) p(z|y)] (18) + log p(y) b sinkhorn's algorithm algorithm 1 sinkhorn's algorithm input: probability vectors s and t, the cost matrix c, weight of the entropic regularization term 1/ , tolerance ε; output: optimal transport plan p * ; 1: u ← 1 c /c, v ← 1 k /k, h ← exp(-c/ ); 2: while @xmath0: v ← t./(h t u); 4: u ← s./(hv); 5: t ← v. * (h t u); 6: err ← 1 t k t -t ; 7: end while 8: p * ← diag(u)hdiag(v); c training algorithm algorithm 2 training algorithm for lantm input: observations o, number of epochs n iter ; 1: initialize model parameters θ μ , θ σ , w, t, l; 2: decide a decoder dec(•); 3: for i from 1 to n iter do 4: compute p * by eq. (7); 5: λ ← kp * ; 6: for batch b from o do 7: for (x, y) in b do 8: μ ← f μ (x), σ ← f σ (x); 9: draw r ∼ n (μ, σ) with reparameterization trick; 10: z ← softmax(r + ln λ y,• ); 11: compute l(x, y) by eq. (3); 12: end for 13: compute l by eq. (13); 14: update θ μ , θ σ , w, t, l; 15: end for 16: end for