v ONTSS : v MF based semi-supervised neural topic modeling with optimal transport (2023)

Abstract: Recently, Neural Topic Models (NTM), inspired by variational autoencoders, have attracted a lot of research interest; however, these methods have limited applications in the real world due to the challenge of incorporating human knowledge. This work presents a semi-supervised neural topic modeling method, vONTSS, which uses von Mises-Fisher (vMF) based variational autoencoders and optimal transport. When a few keywords per topic are provided, vONTSS in the semi-supervised setting generates potential topics and optimizes topic-keyword quality and topic classification. Experiments show that vONTSS outperforms existing semi-supervised topic modeling methods in classification accuracy and diversity. vONTSS also supports unsupervised topic modeling. Quantitative and qualitative experiments show that vONTSS in the unsupervised setting outperforms recent NTMs on multiple aspects: vONTSS discovers highly clustered and coherent topics on benchmark datasets. It is also much faster than the state-of-the-art weakly supervised text classification method while achieving similar classification performance. We further prove the equivalence of optimal transport loss and cross-entropy loss at the global minimum.

## introduction topic modeling methods such as @xcite is an unsupervised approach for discovering latent structure in documents and achieving great performance @xcite . topic modeling methods take a list of documents as input. it generates the defined number of topics. it can further produce keywords and related documents for each topic. in recent years, topic modeling methods have been widely used in many fields such as finance @xcite , healthcare @xcite , education @xcite , marketing @xcite ) and social science @xcite . with the development of variational autoencoder (vae) @xcite , neural topic model @xcite has attracted attention as it enjoys better flexibility and scalability. however, recent research @xcite shows that the topics generated by these methods are not aligned with human perceptions. to incorporate users' domain knowledge into the model, semi-supervised topic modeling methods become an active area of research @xcite @xcite and applications @xcite @xcite . semi-supervised topic modeling methods take a few keywords as input and generate topics based on these keywords. poeple use semisupervised topic modeling methods because they want each topic include certain keywords and incorporate their domain expertise in their generated topics. traditional semi-supervised topic modeling methods fail to utilize semantic information of the corpus, causing low classification accuracy and high variance @xcite . to solve these problems, we propose a von mises-fisher(vmf) based semi-supervised neural topic modeling method using optimal transport (vontss). we use the encoder-decoder framework for our model. the encoder uses modified vmf priors for latent distributions. the decoder uses a word-topic similarity matrix based on spherical embeddings. we use optimal transport to extend it to a semi-supervised version. vontss has the following enhancements: ntm usually utilizes a neural network with softmax to approximate p θ (x|z) := sof tmax(w z) @xcite . ntm selects gaussian @xcite , gamma @xcite and dirichlet distribution @xcite to approximate p(z). the second term kullback-leibler (kl) divergence regularizes q ϕ (z|x) to be close to p(z). ntm has several problems in practice. firstly, it does not capture the semantic relationship between words. secondly, the generated topics are not aligned with human interpretations. @xcite . thirdly, using gaussian prior may risk gravitating latent space toward the center and produce tangled representations among classes of documents. this is due to the fact that gaussian density presents a concentrated mass around the origin in low dimensional settings @xcite and resembles a uniform distribution in high dimensional settings. extending ntm to semi-supervised version is also challenging. l θ,ϕ is not always aligned with classification-related loss such as cross-entropy loss as identified by existing research @xcite . to be specific, cross-entropy makes keywords sets align with assigned topics, while reconstruction loss(-e q ϕ (z|x) [log p θ (x|z)]) makes latent space as representative as possible. thus, existing semi-supervised ntm methods either are not stable @xcite or need certain adaptions @xcite . embedding topic model (etm) pre-trained word embeddings such as glove @xcite and word2vec @xcite have the ability to capture semantic information, which is missing from basic bag-of-word (bow) representations. they can serve as additional information to guide topic discovery. dieng @xcite proposes etm to use a vocabulary embedding matrix e @xmath0. the decoder ϕ learns a topic embedding matrix e @xmath1. we denote topic to word distribution sof tmax(e however since there exists some common words that are related to many other words, these common words' embeddings may be highly correlated with few topics' embeddings. thus, etm does not produce diverse topics @xcite . besides, using pre-trained embeddings cannot help the model identify domain-specific topics. for example, topics related to covid-19 are more likely to be expressed by a few topics instead of one single topic using pre-trained glove embeddings @xcite since covid-19 is not in the embeddings. von mises-fisher in low dimensions, the gaussian density presents a concentrated probability mass around the origin. this is problematic when the data is partitioned into multiple clusters. an ideal prior should be non-informative and uniform over the parameter space. thus, the von mises-fisher(vmf) is used in vae. vmf is a distribution on the (m-1)-dimensional sphere in r m , parameterized by µ ∈ r m where ||µ|| = 1 and a concentration parameter κ ∈ @xmath2. the probability density function of the vmf distribution for @xmath3: ## proposed methods the architecture of vontss is shown in figure 3 . at a high level, our encoder network ϕ transforms the bow representation of the document x d into a latent vector generated by vmf distribution and generates a sample η d . we then apply a temperature function τ and softmax on this sample to get a probabilistic topic distribution z d . lastly, our decoder uses a modified topic-word matrix e to reconstruct x d 's bow representation. to extend into semi-supervised setting, we leverage optimal transport to match keywords' set with topics. the encoder network ϕ and generative model parameter θ are learned jointly during the training process. to overcome entangled topic latent space introduced by gaussian distribution and limited expressibility of vmf distribution, we make two improvements: 1. introduce a temperature function τ (η i ) prior to sof tmax() to modify the radius of vmf distribution. 2. set κ to a learnable parameter to flexibly infer the confidence of particular topics during training. encoder network temperature function to alleviate concerns regarding expressibility while inducing separability among topics, we modify the radius of vmf distribution. we use a temperature function to represent the radius. as shown in figure 1 , unmodified vmf distribution has limited expressiveness. for instance, gaussian posteriors can express a topic probability vector of [0.98, 0.01, 0.005, 0.0003, 0.0002], while vmf can't due to the unity constraint. in practice, if we change the radius to 10, the network can learn more polarized topics distribution as shown in the right plot in figure 1 . the influence of different radii is analyzed in appendix n. given equally powerful learning networks of distributions' parameters, vmf with different radii learns richer and more nuanced structures in their latent representations than a gaussian counterpart (appendix i). learnable κ to further improve the clusterability, we convert κ from a fixed value to a learnable parameter. the kl divergence of vmf distribution makes the distribution more concentrated while not influencing the direction of latent distribution. this makes the result more clustered. for gaussian distribution, kl divergence penalizes the polarization of latent distribution (appendix k). this makes the gaussian distribution less clustered. to illustrate this, we randomly sampled encoded documents' latent distributions from agnews dataset @xcite after training with both latent distributions, as shown in figure 2 . for the gaussian distribution, we see that documents belonging to different topics are entangled around the center, causing the inseparability of topics during both the training and inference stage. vmf distribution, on the hand, repels four document classes into different quadrants, presents more structures when compared to gaussian distribution, and creates better separable clusters. detailed ablation study can be found in appendix o decoder network our decoder follows etm's construction and uses the embedding e v and e t to generate a topic-word matrix e. one distinction between our decoder and etm's decoder is that we generate the word embeddings by training a spherical embedding on the dataset. spherical embeddings perform well in word similarity evaluation and document clustering (meng et al., 2019), which further improves the clusterability of the topic modeling methods. we also keep word embeddings fixed during the topic modeling training process for two reasons. firstly, keeping word embeddings fixed can alleviate sparsity issues @xcite . additionally, vmf based vae tends to be less expressive in high dimensions due to limited variance freedom @xcite . keeping the embedding fixed can make topics more separable in higher-dimension settings and improve topic diversity. ## conclusions in this paper, we propose a new semi-supervised neural topic modeling method vontss, which leverages vmf, the temperature function, optimal transport, and vaes. its unsupervised version exceeds state-of-the-art in topic coherence through both unsupervised and human evaluations while inducing high clusterability among topics. we show that optimal transport loss is equivalent to crossentropy loss under the optimal condition and induces one-to-one mapping between keywords sets and topics. vontss achieves competitive classification performance, maintains top topic diversity, trains fast, and possesses the least variance among diverse datasets. step 1 show that p t,@xmath4, s) ∈ (t, s) and equal to 0 in all other cases. ∃p t,@xmath5, s) ∈ (t, s). without loss of generality, we assume p t,s ′ = 1γ, p t ′ ,s ′ = γ and p t ′ ,@xmath6. consider related term in l ot , for the first term: γ(c t,s + c t ′ ,s ′ ) + (1γ)(c t,s ′ + c t ′ ,s ) = (c t,s + c t ′ ,s ′ ) -(1γ)(c t,s + c t ′ ,s ′ -(c t,s ′ + c t ′ ,s )) ≥ c t,s + c t ′ ,s ′ using lemma 3.1 and equation ( 7 ) for the second term in l ot , -p t,s log p t,@xmath7,@xmath8. otherwise, it is larger than 0. this means that p t,@xmath9,s ′ = 1 achieve smaller l ot compare to current settings. this contradicts the definition of l ot which is the min in the space. thus, p t,@xmath10, s) ∈ (t, s). since the raw sum and column sum equal to |t |. this means p t,@xmath11, s) / ∈ (t, s) 10 ) and ( 11 ), we have l @xmath12,s)∈(s,t ) c t,sh(p