N euro M ax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization (2024)

Abstract: Recent advances in neural topic models have concentrated on two primary directions: the integration of the inference network (encoder) with a pre-trained language model (PLM) and the modeling of the relationship between words and topics in the generative model (decoder). However, the use of large PLMs significantly increases inference costs, making them less practical for situations requiring low inference times. Furthermore, it is crucial to simultaneously model the relationships between topics and words as well as the interrelationships among topics themselves. In this work, we propose a novel framework called NeuroMax (**Neur**al T**o**pic Model with **Max**imizing Mutual Information with Pretrained Language Model and Group Topic Regularization) to address these challenges. NeuroMax maximizes the mutual information between the topic representation obtained from the encoder in neural topic models and the representation derived from the PLM. Additionally, NeuroMax employs optimal transport to learn the relationships between topics by analyzing how information is transported among them. Experimental results indicate that NeuroMax reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings, thereby enhancing performance on downstream tasks.

## introduction topic modeling @xcite @xcite @xcite @xcite ) is a well-established task in natural language processing (nlp) that involves uncovering and extracting latent topics from extensive corpora, thereby facilitating the comprehension and organization of unstructured data (kherwa and bansal, * equally contributed. regarding the encoder, several studies have proposed incorporating knowledge from pretrained language models @xcite such as bert @xcite and gpt @xcite . these models, trained on vast amounts of text data, effectively capture linguistic patterns and contextual information. this rich information can serve as input for the encoder @xcite , enhancing the topic models' ability to generate coherent topics. however, despite such an advantage, utilizing large pretrained models significantly increases inference costs, which limits their utility in scenarios requiring low inference time. concerning the decoder, a line of work has leveraged pretrained word embeddings to better capture the semantics of the vocabulary @xcite @xcite @xcite . recently, @xcite decomposed the topic-word distribution matrix into word and topic embeddings, with the word embeddings initialized by pretrained knowledge. the topic-word distribution is then modeled as the soft-max of the negative l 2 distance between corresponding embeddings. additionally, their method employs clustering regularization to group word embeddings into clusters, with each cluster corresponding to a topic, thereby mitigating the topic collapse problem. while these approaches improve efficiency in modeling word-topic relationships, they lack adequate consideration for capturing semantic connections between topics, resulting in a topic embedding space that is difficult to interpret. in response to these shortcomings, we propose a neural topic model framework that leverages the power of pretrained knowledge without incurring expensive inference costs and effectively captures semantic interrelationships at the topic level. first, for the encoder, we hypothesize that the topic proportions and embeddings derived from pretrained language models should exhibit similar representational characteristics. by maximizing the mutual information between these two variables, we integrate contextualized information from pretrained language models during training, thereby eliminating the need for pretrained components at the inference stage. figure 1 presents the overall architecture of our proposed encoder, which operates without pretrained language models during inference. second, inspired by @xcite , we employ optimal transport (ot) to model the relationships between topics. specifically, in line with standard works @xcite , we assume that each topic carries an equal amount of information, which is transported from one topic to another in a manner that preserves the total information within each topic. the learned transport plan elucidates the connections between the topics. additionally, we assume that documents often encompass several closely related themes, naturally grouping topics into semantically related clusters. we enhance the group relationship of topics by imposing a regularization on the aforementioned transport plan based on predefined topic cluster relationships derived from topic clustering. the rest of this paper is organized as follows: section 2 lists some related works in topic modeling and neural topic modeling. some background on neural topic models, mutual information maximization, and optimal transport is provided in 3. our proposed methodologies are introduced in 4. some experiments to illustrate the effectiveness of the proposed method are reported in 5. finally, our discussion and conclusion are given in section 6. 2 related work topic models and neural topic models. traditionally, generative probabilistic models such as latent dirichlet allocation (lda) @xcite and probabilistic latent semantic indexing (plsi) @xcite have been utilized for topic modeling. numerous extensions of these models have been proposed to accommodate various assumptions and settings @xcite @xcite @xcite @xcite . recent advancements have integrated topic models with variational autoencoders (vae) (kingma and welling, 2013) to improve scalability and efficiency in the inference process @xcite @xcite @xcite @xcite . due to the difficulty of sampling the dirichlet prior using the reparameterization trick, a laplacian approximation is utilized in @xcite ). an alternative method involves using rejection sampling variational inference for the dirichlet prior @xcite . within vae architecture, two lines of research aim at improving the two components of the model, the encoder and the decoder. in both directions, incorporating external knowledge like word embeddings has become a prevalent practice to enhance topic quality @xcite @xcite @xcite . regarding the decoder (or the reconstruction phase of topic modeling), (dieng et al., 2020) proposed using word embeddings, such as word2vec @xcite or glove @xcite , to gain a better understanding of vocabulary semantics, thus creating topics with more semantically related words. various variants of word embedding are considered; for example, @xcite utilized spherical embeddings to improve clusterability, while @xcite employed word embeddings in hyperbolic space for topic taxonomy. in the embedding space, dependency between words and topics is modeled as a similarity function, such as dot product @xcite , cosine similarity @xcite , or exponent of negative l 2 distance @xcite . additionally, @xcite applied a clustering regularization technique to ensure that each topic embedding acts as the center of a distinct cluster of word embeddings, thereby mitigating the issue of topic collapse. in terms of encoders, pretrained language models @xcite are another type of external knowledge frequently incorporated. @xcite used contextualized document embedding from sbert @xcite as input, capturing valuable information in complement to bagof-word representation. @xcite further employed sbert to generate term weights that are integrated into the reconstruction loss to filter out irrelevant words. these approaches lead to increased inference time for neural topic models, which poses challenges for real-time applications. an improvement that does not rely on external knowledge involves utilizing optimal transport distance to model the disparity between documents and topics @xcite . along with the development of pretrained language models, an alternate line of research in topic modeling that does not employ a vae-like architecture directly group document's embedding to generate topics @xcite @xcite . although this approach is simpler and yields coherent topics, it is not trivial to infer the topic proportions for a document. ## proposed method we enhance both the inference network (encoder) and the generative model (decoder) of neural topic models through mutual information maximization and group topic regularization, respectively. the details will be presented in the subsequent subsections. ## conclustion in conclusion, this paper introduces neuromax, a novel framework designed to tackle critical challenges in neural topic modeling. by maximizing the mutual information between the topic representation obtained from the common encoder in neural topic models and the representation derived from the plm and leveraging optimal transport to capture topic relationships, neuromax offers a comprehensive solution for improving topic modeling efficiency and quality. our experimental results demonstrate that neuromax significantly reduces inference time and obtains more coherent topics and topic groups, thus enhancing document representation for downstream task effectiveness. with its innovative approach and promising results, neu-romax represents a valuable contribution to the field of neural topic modeling.