D e T i ME : Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM (2023)

Abstract: In the burgeoning field of natural language processing, Neural Topic Models (NTMs) and Large Language Models (LLMs) have emerged as areas of significant research interest. Despite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic generation. Our study addresses this gap by introducing a novel framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME). DeTiME leverages Encoder-Decoder-based LLMs to produce highly clusterable embeddings that could generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods. Additionally, by exploiting the power of diffusion, our framework also provides the capability to generate content relevant to the identified topics. This dual functionality allows users to efficiently produce highly clustered topics and related content simultaneously. DeTiME’s potential extends to generating clustered embeddings as well. Notably, our proposed framework proves to be efficient to train and exhibits high adaptability, demonstrating its potential for a wide array of applications.

## introduction topic modeling methods, such as @xcite , are unsupervised approaches for discovering latent structures in documents and achieving great performance @xcite . these methods take a list of documents as input, generate a defined number of topics, and can further produce keywords and related documents for each topic. in recent years, topic modeling methods have been widely used in various fields such as finance @xcite , healthcare @xcite , education @xcite , marketing @xcite , and social science @xcite . with the development of variational autoencoder (vae) @xcite , the neural topic model @xcite has attracted attention due to its better flexibility and scalability. the topic is generated through the reconstruction of the bag-of-word representations of the document @xcite . the progress of large language model (llm) @xcite brings significant advancements in the nlp community. sentence embedding is the process of converting sentences into numerical vectors in a high-dimensional space. llm-based sentence embedding has been applied to topic modeling by using it to reconstruct bag of word representation of documents @xcite , to cluster document directly @xcite or both @xcite . sentence embedding-based models have been shown to achieve high performance regarding coherence and diversity @xcite . embeddings with higher clusterability are likely to perform well in classification tasks. however, sentence embeddings are in general not perform well in clustering. the best performed sentence embedding has an average v-measure @xcite below 0.44 even if it uses kmeans and set the cluster equal to the number of different labels @xcite . this means that their clusterability can be even lower when the latent dimension increases. lastly, language modeling is a powerful generative tool @xcite . while topic modeling has been utilized for generation @xcite , its integration with large language models (llms) for generation remains less explored. in this study, we introduce detime, an innovative topic modeling framework that exploits the capabilities of the encoder-decoder large language model (llm). specifically, we design a task to train an adapted encoder-decoder llm, as depicted in figure 2 . we generate an embedding using this architecture, which exhibits high clusterability compared to established models as illustrated in figure 1 . furthermore, we design a topic modeling approach using the last hidden layer of our modified llm encoder as input. this technique notably outperforms standard methods across all pertinent metrics. additionally, we leverage diffusion and our proposed framework to generate relevant documents. our major contributions are as follows: by documenting detailed methodology and empirical results, we aim to inspire further research in this domain, and provide a strong foundation for future work on topic modeling and llms. ## methods the goal of this paper is to create a framework that leverages encoder-decoder llm to generate topics that is highly clusterable and able to generate topic related sentence. to achieve that, we need to create an embedding that could be used to generate text as well as be highly clusterable. thus, we designed a specific task and dataset for our use case. we add cnn encoder and decoder on top of flant5 to generate that can easily fit into neural topic modeling for further dimension reduction. we further design a variational autoencoder to take the output of the cnn encoder as input and generate topics and reconstruct embeddings. this is achieved by 2 autoencoders. the first autoencoder is a variational autoencoder which generates topic distribution and reconstructs bag of words representations. to reconstruct the embeddings from enc 2 . we use another autoencoder to generate embeddings from topic distribution and reconstructed bag of words. the detailed structure and name are in figure 3 . we do not train or finetune flant5 and cnn during the topic modeling process which makes our methods cost-effective. we then leverage diffusion to generate high quality text that represents the document. this section contains four components. first, we present the dataset and define the finetuned task. second, we elaborate on our modified flant5 and the fine-tuning strategy. the third component introduces a variational autoencoder designed for topic modeling and generation. finally, we utilize diffusion to generate content relevant to the derived topics. ## conclusion and future work we have developed a framework detime for generating highly clusterable embeddings, leveraging the strengths of paraphrase tasks, flant5, and cnn. in addition to this, we introduced a variational autoencoder structure capable of reconstructing embeddings while simultaneously producing highly coherent, diverse, and clusterable topics. our design incorporates a diffusion process for generating content that provides representative depictions of various topics. the flexibility of our embedding generation structure allows for easy adaptation to other encoder-decoder language model architectures, eliminating the need for retraining the entire framework, thereby ensuring cost-effectiveness. additionally, our variational autoencoder structure is versatile, and capable of being applied to any contextual embeddings. other methods could further improve with larger llm. moving forward, we aim to further improve the performance of our embeddings by training on larger models such as flan-t5-xl. benchmarking other pre-training with fine-tuning (peft) methods, such as lora, may also enhance our system's performance. given the high clusterability of our embeddings, we plan to extend our work to semi-supervised document classification @xcite @xcite . this framework could be applied to identify the most representative documents within extensive document collections. this functionality could make our model suitable for generation topic guided generation @xcite ) finally, we envisage utilizing this framework to generate superior summarizations for large documents. this could be achieved by training a decoder for summarization, generating a summarization for each topic, and subsequently concatenating them. this framework can also be extended to hierarchical topic modeling @xcite @xcite , mitigate data sparsity for short text topic modeling @xcite , generate topic-relevant and coherent long texts @xcite , and construct a network of topics together with meaningful relationships between them @xcite .